Capturing and processing of images including occlusions focused on an image sensor by a lens stack array

ABSTRACT

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. An imaging device in accordance with one embodiment of the invention includes at least one imager array, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements, control circuitry configured to capture images formed on the light sensing elements of each of the imagers, and a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured images.

RELATED APPLICATIONS

The current application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 14/943,009 entitled “Capturing and Processing ofImages Using Camera Array Incorporated Bayer Cameras Having DifferentFields of View”, filed Nov. 16, 2015 and issued on Aug. 29, 2017 as U.S.Pat. No. 9,749,547, which is a continuation of U.S. Non-Provisionalpatent application Ser. No. 14/704,920, entitled “Capturing andProcessing of Images Including Occlusions Focused on an Image Sensor bya Lens Stack Array”, filed May 5, 2015 and issued on Nov. 17, 2015 asU.S. Pat. No. 9,188,765, which application is a continuation of U.S.Non-Provisional patent application Ser. No. 14/459,288, entitled“Systems and Methods for Normalizing Image Data Captured by CameraArrays”, filed Aug. 13, 2014 and issued on Jun. 2, 2015 as U.S. Pat. No.9,049,381, which application is a continuation of U.S. Non-Provisionalpatent application Ser. No. 12/952,134, entitled “Capturing andProcessing of Images Using Monolithic Camera Array with HeterogeneousImagers”, filed Nov. 22, 2010 and issued on Oct. 21, 2014 as U.S. Pat.No. 8,866,920, which claims priority to U.S. Provisional PatentApplication 61/281,662 filed Nov. 20, 2009 and U.S. Provisional PatentApplication 61/263,339 filed Nov. 20, 2009, the disclosures of which arehereby incorporated by reference in their entirety. U.S. Non-Provisionalpatent application Ser. No. 12/952,134 also claims priority as aContinuation-In-Part application to U.S. patent application Ser. No.12/935,504, entitled “Capturing and Processing of Images UsingMonolithic Camera Array With Heterogeneous Imagers”, filed Sep. 29, 2010and issued on Dec. 2, 2014 as U.S. Pat. No. 8,902,321, which is anational stage application of Patent Cooperation Treaty ApplicationPCT/US2009/044687 filed May 20, 2009, and claims priority to U.S.Provisional Patent Application 61/054,694, filed May 20, 2008, thedisclosures of which hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present invention is related to an image sensor including aplurality of heterogeneous imagers, more specifically to an image sensorwith a plurality of wafer-level imagers having custom filters, sensorsand optics of varying configurations.

BACKGROUND

Image sensors are used in cameras and other imaging devices to captureimages. In a typical imaging device, light enters through an opening(aperture) at one end of the imaging device and is directed to an imagesensor by an optical element such as a lens. In most imaging devices,one or more layers of optical elements are placed between the apertureand the image sensor to focus light onto the image sensor. The imagesensor consists of pixels that generate signals upon receiving light viathe optical element. Commonly used image sensors include CCD(charge-coupled device) image sensors and CMOS (complementarymetal-oxide-semiconductor) sensors.

Filters are often employed in the image sensor to selectively transmitlights of certain wavelengths onto pixels. A Bayer filter mosaic isoften formed on the image sensor. The Bayer filter is a color filterarray that arranges one of the RGB color filters on each of the colorpixels. The Bayer filter pattern includes 50% green filters, 25% redfilters and 25% blue filters. Since each pixel generates a signalrepresenting strength of a color component in the light and not the fullrange of colors, demosaicing is performed to interpolate a set of red,green and blue values for each image pixel.

The image sensors are subject to various performance constraints. Theperformance constraints for the image sensors include, among others,dynamic range, signal to noise (SNR) ratio and low light sensitivity.The dynamic range is defined as the ratio of the maximum possible signalthat can be captured by a pixel to the total noise signal. Typically,the well capacity of an image sensor limits the maximum possible signalthat can be captured by the image sensor. The maximum possible signal inturn is dependent on the strength of the incident illumination and theduration of exposure (e.g., integration time, and shutter width). Thedynamic range can be expressed as a dimensionless quantity in decibels(dB) as:

$\begin{matrix}{{DR} = \frac{{full}\mspace{14mu}{well}\mspace{14mu}{capacity}}{{RMS}\mspace{14mu}{noise}}} & {{equation}\mspace{14mu}(1)}\end{matrix}$Typically, the noise level in the captured image influences the floor ofthe dynamic range. Thus, for an 8 bit image, the best case would be 48dB assuming the RMS noise level is 1 bit. In reality, however, the RMSnoise levels are higher than 1 bit, and this further reduces the dynamicrange.

The signal to noise ratio (SNR) of a captured image is, to a greatextent, a measure of image quality. In general, as more light iscaptured by the pixel, the higher the SNR. The SNR of a captured imageis usually related to the light gathering capability of the pixel.

Generally, Bayer filter sensors have low light sensitivity. At low lightlevels, each pixel's light gathering capability is constrained by thelow signal levels incident upon each pixel. In addition, the colorfilters over the pixel further constrain the signal reaching the pixel.IR (Infrared) filters also reduce the photo-response from near-IRsignals, which can carry valuable information.

These performance constraints of image sensors are greatly magnified incameras designed for mobile systems due to the nature of designconstraints. Pixels for mobile cameras are typically much smaller thanthe pixels of digital still cameras (DSC). Due to limits in lightgathering ability, reduced SNR, limits in the dynamic range, and reducedsensitivity to low light scenes, the cameras in mobile cameras show poorperformance.

SUMMARY

Camera arrays, imaging devices including a camera array and/or a methodfor capturing an image that employ a plurality of imagers where eachimager includes a plurality of sensor elements, and lens stack arraysthat can be utilized in camera arrays in accordance with embodiments ofthe invention are disclosed. The plurality of imagers can include atleast a first imager and a second imager, where the first imager and thesecond imager may have the same imaging characteristics or differentimaging characteristics.

In one embodiment, the first imaging characteristics and the secondimager have different imaging characteristics. The imagingcharacteristics may include, among others, the size of the imager, thetype of pixels included in the imager, the shape of the imager, filtersassociated with the imager, exposure time of the imager, aperture sizeassociated with the imager, the configuration of the optical elementassociated with the imager, gain of the imager, the resolution of theimager, and operational timing of the imager.

In one embodiment, the first imager includes a filter for transmitting alight spectrum. The second imager also includes the same type of filterfor transmitting the same light spectrum as the first imager butcaptures an image that is sub-pixel phase shifted from an image capturedby the first imager. The images from the first imager and the secondimager are combined using a super-resolution process to obtain images ofhigher resolution.

In one embodiment, the first imager includes a first filter fortransmitting a first light spectrum and the second imager includes asecond filter for transmitting a second light spectrum. The images fromthe first and second imagers are then processed to obtain a higherquality image.

In one embodiment, lens elements are provided to direct and focus lightonto the imagers. The lens elements form lens stacks that create opticalchannels, and each lens stack focuses light onto one imager. Becauseeach lens element is associated with one imager, each lens element maybe designed and configured for a narrow light spectrum. Further, thethickness of the lens element may be reduced, decreasing the overallthickness of the camera array. In such an embodiment, the lens elementsmay be made using any suitable fabrication technique, such as, forexample using wafer level optics (WLO) technology, injection molding,and/or glass molding.

In one embodiment, the plurality of imagers include at least one near-IRimager dedicated to receiving near-IR (Infrared) spectrum. An imagegenerated from the near-IR imager may be fused with images generatedfrom other imagers with color filters to reduce noise and increase thequality of the images. In another such embodiment, imagers coveringother spectrum ranges including far-IR and UV spectra may be included aswell.

In one embodiment, the plurality of imagers may be associated with lenselements that provide a zooming capability. In one such embodiment,different imagers may be associated with lenses of different focallengths to have different fields-of-views and provide different levelsof zooming capability. Different fields-of-view can also be achievedusing imagers having different sensor sizes/formats by means ofdifferent pixel sizes or different numbers of pixels/light sensingelements. A mechanism may be provided to provide smooth transition fromone zoom level to another zoom level.

In one or more embodiments, the plurality of imagers is coordinated andoperated to obtain at least one of a high dynamic range image, apanoramic image, a hyper-spectral image, distance to an object and ahigh frame rate video.

An imaging device in accordance with one embodiment of the inventionincludes at least one imager array, and each imager in the arraycomprises a plurality of light sensing elements and a lens stackincluding at least one lens surface, where the lens stack is configuredto form an image on the light sensing elements, control circuitryconfigured to capture images formed on the light sensing elements ofeach of the imagers, and a super-resolution processing module configuredto generate at least one higher resolution super-resolved image using aplurality of the captured images.

In a further embodiment of the invention, a spectral filter is providedwithin each imager and is configured to pass a specific spectral band oflight, and spectral filters that pass different spectral bands areprovided within at least two of the imagers.

In another embodiment of the invention, the spectral filter is selectedfrom the group consisting of an organic color filter, an absorptivematerial, a dielectric coating, an interference filter, a multilayercoating, and combinations thereof.

In a still further embodiment of the invention, at least one of theimagers further comprises a polarizing filter.

In still another embodiment of the invention, the construction of thelens stack within an imager differs based upon the specific spectralband of light passed by the spectral filter within the imager such thatchromatic aberrations are reduced.

In a yet further embodiment of the invention, the prescription of atleast one surface of the lens stack within an imager is a function ofthe specific spectral band of light passed by the spectral filter withinthe imager.

In yet another embodiment of the invention, the at least one surface ofthe lens stack is configured so that the back focal length of each lensstack in the imager array are the same independent of the spectral band.

In a further embodiment again of the invention, a combination of highand low Abbe number materials is used in the construction of each of thelens stacks to reduce chromatic aberrations.

In another embodiment again of the invention, each lens stack includesat least one aperture stop positioned in front of the light sensingelement, and the spectral filter is positioned in proximity to theaperture stop.

In a further additional embodiment of the invention, the at least oneaperture stop is formed by a light blocking material selected from thegroup consisting of metal materials, oxide materials, black particlefilled photoresists and combinations thereof.

In another additional embodiment of the invention, at least one lenssurface in the optical stack of an imager differs based upon thespecific spectral band of light passed by the spectral filter within theimager, and each lens surface is selected from the group consisting ofdiffractive, Fresnel, refractive and combinations thereof.

In a still yet further embodiment of the invention, the radii ofcurvature of the lens surfaces differ based upon the specific spectralband of light passed by the spectral filter within the imager.

In still yet another embodiment of the invention, the lens stacks ofeach of the imagers have the same back focal length.

In a still further embodiment again of the invention, the light sensingelements are disposed to detect an image in a backside imaging mode.

In still another embodiment again of the invention, a separate negativelens element is disposed in close proximity to the light sensingelements of an imager and in optical alignment with the lens stack ofthe imager so that the field curvature of the lens stack is corrected.

In a still further additional embodiment of the invention, a separatenegative lens element is disposed in close proximity to the lightsensing elements of an imager and in optical alignment with the lensstack of the imager so that the field curvature of the lens stack iscorrected.

In still another additional embodiment of the invention, at least two ofthe imagers in the imager array have different fields of view so thatdifferent scene sizes can be captured by the imagers.

In a yet further embodiment again of the invention, the lens stacks ofthe at least two imagers have different focal lengths.

In yet another embodiment again of the invention, the light sensingelements in each of at least two imagers have different sizes.

In a yet further additional embodiment of the invention, the at leasttwo imagers include different numbers of light sensing elements.

Yet another additional embodiment of the invention also includes amechanical zoom mechanism within the lens stack of an imager forsmoothly transitioning between different fields of view.

In a further additional embodiment of the invention again, at least oneof the imagers is designed to image light in the near-infrared spectrum.

In another additional embodiment of the invention again, at least oneimager is configured to take a measurement of the visible lightspectrum, and wherein the control circuitry is configured to use saidmeasurement to control the imaging characteristics of the other imagersin said imaging device.

In a still yet further additional embodiment of the invention, theimaging device includes at least two camera arrays.

In still yet another additional embodiment of the invention, the atleast two camera arrays operate in conjunction during imaging to providestereo images.

In a still yet further embodiment of the invention again, the at leasttwo camera arrays are spatially separated one from the other in at leastone plane.

In still yet another embodiment of the invention again, each of thecamera arrays include a different combination of imagers eachcombination of imagers having different imaging characteristics.

In a yet further additional embodiment of the invention again, at leastone camera array comprises an N×M array of imagers, where at least oneof N and M is greater than 2, and each of said imagers comprising atleast two light sensing elements.

In yet another additional embodiment of the invention again, the controlcircuitry is configured to control the imaging characteristics of eachimager.

In another further embodiment of the invention, the controllable imagingcharacteristics include exposure time, gain, and black level offset.

In still another further embodiment of the invention, the controlcircuitry is configured to trigger each of said imagers in accordancewith a desired sequence.

In yet another further embodiment of the invention, the controlcircuitry operates the imagers in a staggered sequence.

In another further additional embodiment of the invention, an initialmaster setting for the imaging characteristics of each imager is set bythe control circuitry, predefined deviations from the initial mastersettings are also set by the control circuitry, and the deviationsinclude functions selected from the group consisting of ISO settings,high dynamic range, gain settings, integration time settings, exposuresettings, digital processing settings and combinations thereof.

In another further embodiment of the invention again, the camera arraysare fabricated using techniques consisting of wafer level opticstechniques, injection molding, glass molding and combinations thereof.

In still yet another further embodiment of the invention, each of theplurality of imagers are optically separated from each other.

In still another further additional embodiment of the invention, each ofthe imagers is optically separated from the other imagers by at leasttwo opaque surfaces located in front of the light sensing elements andhaving openings arranged in axial alignment with the lens stack of theimager.

In still another further embodiment of the invention again, the imagersare optically separated from each other by opaque walls disposed at theboundaries between the imagers.

In yet another further additional embodiment of the invention, theopaque walls are cavities between the imagers filled with light blockingmaterial.

In yet another further embodiment of the invention again, the lens stackincludes spacers, and the spacers are constructed from an opaquematerial.

In another further additional embodiment of the invention again, thelens stack includes spacers, and the spacers are coated with an opaquematerial.

In still yet another further additional embodiment of the invention, thecontrol circuitry is configured to perform normalization of a capturedlow resolution image using a one dimensional space filling curve todescribe a normalization plane.

In still yet another further embodiment of the invention again, the onedimensional space filling curve is expressed as the coefficients of apolynomial.

In yet another further additional embodiment of the invention again, thecontrol circuitry is configured to capture multiple low resolutionimages using different imagers having different exposure times.

In still yet another further additional embodiment of the inventionagain, the light sensing elements of at least one of the imagers used tocapture an image are high dynamic range light sensing elements.

In an additional embodiment of the invention, at least one of theimagers used to capture a low resolution image is configured to capturea near-IR low resolution image.

In an additional embodiment of the invention again, the super-resolutionmodule is configured to generate a depth map for the super-resolutionimage.

In a number of embodiments of the invention, the super-resolution moduleis configured to select at least one distance as a focal plane and toapply blurring to pixels with depths that are not proximate a focalplane.

In a further embodiment of the invention, the super-resolutionprocessing module is configured to perform temperature normalizationwith respect to the captured images.

In a still further embodiment of the invention, the point spreadfunctions of the lens stacks vary with temperature, and thesuper-resolution processing module is configured to perform temperaturenormalization by selecting a point spread functions from temperaturecalibration data corresponding to the point spread functions of the lensstacks during the capture of the images.

In still another embodiment of the invention, the super-resolutionprocessing module is configured to computationally correct colordifferences between the captured images.

In a yet further embodiment of the invention, the super-resolutionprocessing module is configured to computationally correct geometricdistortion differences between the captured images.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a plan view of a camera array with a plurality of imagers,according to one embodiment.

FIG. 2A is a perspective view of a camera array with lens elements,according to one embodiment.

FIG. 2B is a cross-sectional view of a camera array, according to oneembodiment.

FIG. 2C is a cross-sectional view of a camera array with opticalcrosstalk suppression, according to one embodiment.

FIG. 2D is a cross-sectional view of a camera array with opticalcrosstalk suppression, according to a second embodiment.

FIG. 2E is a cross-sectional view of a camera array incorporating opaquespacers to provide optical crosstalk suppression, according to a furtherembodiment.

FIG. 2F is a cross-sectional view of a camera array incorporatingspacers coated with opaque material to provide crosstalk suppression,according to another embodiment.

FIGS. 3A and 3B are sectional diagrams illustrating changes in theheights of lens elements depending on changes in the dimensions ofimagers, according to one embodiment.

FIG. 3C is a diagram illustrating chief ray angles varying depending ondiffering dimensions of the lens elements.

FIG. 3D is a cross-sectional view of a camera array with fieldflattening, according to one embodiment.

FIG. 4 is a functional block diagram for an imaging device, according toone embodiment.

FIG. 5 is a functional block diagram of an image processing pipelinemodule, according to one embodiment.

FIGS. 6A through 6F are plan views of camera arrays having differentlayouts of heterogeneous imagers, according to embodiments.

FIG. 6G is a diagram conceptually illustrating the manner in whichsampling diversity can depend upon object distance.

FIG. 6H is a is a cross sectional view of pixels of an imager inaccordance with an embodiment of the invention

FIG. 6I is a diagram conceptually illustrating occlusion zones createdwhen Red and Blue imagers are not symmetrically distributed about thecentral access of a camera array.

FIG. 6J is a diagram conceptually illustrating the manner in which theocclusion zones illustrated in FIG. 6I are eliminated by distributingRed and Blue imagers symmetrically about the central access of a cameraarray.

FIG. 7 is a flowchart illustrating a process of generating an enhancedimage from lower resolution images captured by a plurality of imagers,according to one embodiment.

FIG. 7A seems to be a flowchart is a flow chart illustrating a processfor constructing a normalization plane during calibration in accordancewith an embodiment of the invention.

FIG. 7B conceptually illustrates the process for constructing anormalized plane during calibration in accordance with an embodiment ofthe invention illustrated in FIG. 7A.

FIG. 8A is a cross-sectional view of a camera array with optical zoom,according to one embodiment.

FIG. 8B is a cross-sectional view of a camera array with optical zoom,according to a second embodiment.

FIG. 8C is a cross-sectional view of a camera array with imagers havingdifferent fields-of-view, according to a further embodiment.

DETAILED DESCRIPTION

Embodiments of the invention are now described with reference to thefigures where like reference numbers indicate identical or functionallysimilar elements. Also in the figures, the left most digits of eachreference number corresponds to the figure in which the reference numberis first used.

Many embodiments relate to using a distributed approach to capturingimages using a plurality of imagers of different imagingcharacteristics. Each imager may be configured in such a manner thateach imager captures an image that is shifted by a sub-pixel amount withrespect to the image captured by other imagers having similar imagingcharacteristics. Each imager may also include separate optics withdifferent filters and operate with different operating parameters (e.g.,exposure time). Distinct images generated by the imagers are processedto obtain an enhanced image. In many embodiments, the separate opticsincorporated into each imager are implemented using a lens stack array.The lens stack array can include one or more optical elements fabricatedusing wafer level optics (WLO) technology.

A sensor element or pixel refers to an individual light sensing elementin an imager. The light sensing element can be, but is not limited to,traditional CIS (CMOS Image Sensor), CCD (charge-coupled device), highdynamic range pixel, multispectral pixel and various alternativesthereof.

A sensor refers to a two dimensional array of pixels used to capture animage formed on the sensor by the optics of the imager. The sensorelements of each sensor have similar physical properties and receivelight through the same optical component. Further, the sensor elementsin the each sensor may be associated with the same color filter.

A camera array refers to a collection of imagers designed to function asa unitary component. The camera array may be fabricated on a single chipfor mounting or installing in various devices.

An array of camera arrays refers to an aggregation of two or more cameraarrays. Two or more camera arrays may operate in conjunction to provideextended functionality over a single camera array, such as, for example,stereo resolution.

Image characteristics of an imager refer to any characteristics orparameters of the imager associated with capturing of images. Theimaging characteristics may include, among others, the size of theimager, the type of pixels included in the imager, the shape of theimager, filters associated with the imager, the exposure time of theimager, aperture size associated with the imager, the configuration ofthe optical element associated with the imager (such as the number ofelements, the shapes, profiles and sizes of the lens surfaces, includingthe radii of curvature, aspheric coefficients, focal lengths and FOVs ofthe objectives, color correction, F/#s, etc.), the gain of the imager,the resolution of the imager, and operational timing of the imager.

Structure of Camera Array

FIG. 1 is a plan view of a camera array 100 with imagers 1A through NM,according to one embodiment. The camera array 100 is fabricated on asemiconductor chip to include a plurality of imagers 1A through NM. Eachof the imagers 1A through NM may include a plurality of pixels (e.g.,0.32 Mega pixels). In one embodiment, the imagers 1A through NM arearranged into a grid format as illustrated in FIG. 1. In otherembodiments, the imagers are arranged in a non-grid format. For example,the imagers may be arranged in a circular pattern, zigzagged pattern orscattered pattern or an irregular pattern including sub-pixel offsets.

The camera array may include two or more types of heterogeneous imagers,each imager including two or more sensor elements or pixels. Each one ofthe imagers may have different imaging characteristics. Alternatively,there may be two or more different types of imagers where the same typeof imager shares the same imaging characteristics.

In one embodiment, each imager 1A through NM has its own filter and/oroptical element (e.g., lens). Specifically, each of the imagers 1Athrough NM or a group of imagers may be associated with spectral colorfilters to receive certain wavelengths of light. Example filters includea traditional filter used in the Bayer pattern (R, G, B or theircomplements C, M, Y), an IR-cut filter, a near-IR filter, a polarizingfilter, and a custom filter to suit the needs of hyper-spectral imaging.Some imagers may have no filter to allow reception of both the entirevisible spectra and near-IR, which increases the imager'ssignal-to-noise ratio. The number of distinct filters may be as large asthe number of imagers in the camera array. Further, each of the imagers1A through NM or a group of imagers may receive light through lenseshaving different optical characteristics (e.g., focal lengths) orapertures of different sizes.

In one embodiment, the camera array includes other related circuitry.The other circuitry may include, among others, circuitry to controlimaging parameters and sensors to sense physical parameters. The controlcircuitry may control imaging parameters such as exposure times, gain,and black level offset. The sensor may include dark pixels to estimatedark current at the operating temperature. The dark current may bemeasured for on-the-fly compensation for any thermal creep that thesubstrate may suffer from. Alternatively, compensation of thermaleffects associated with the optics, e.g., because of changes inrefractive index of the lens material, may be accomplished bycalibrating the PSF for different temperatures.

In one embodiment, the circuit for controlling imaging parameters maytrigger each imager independently or in a synchronized manner. The startof the exposure periods for the various imagers in the camera array(analogous to opening a shutter) may be staggered in an overlappingmanner so that the scenes are sampled sequentially while having severalimagers being exposed to light at the same time. In a conventional videocamera sampling a scene at N exposures per second, the exposure time persample is limited to 1/N seconds. With a plurality of imagers, there isno such limit to the exposure time per sample because multiple imagersmay be operated to capture images in a staggered manner.

Each imager can be operated independently. Entire or most operationsassociated with each individual imager may be individualized. In oneembodiment, a master setting is programmed and deviation (i.e., offsetor gain) from such master setting is configured for each imager. Thedeviations may reflect functions such as high dynamic range, gainsettings, integration time settings, digital processing settings orcombinations thereof. These deviations can be specified at a low level(e.g., deviation in the gain) or at a higher level (e.g., difference inthe ISO number, which is then automatically translated to deltas forgain, integration time, or otherwise as specified by context/mastercontrol registers) for the particular camera array. By setting themaster values and deviations from the master values, higher levels ofcontrol abstraction can be achieved to facilitate a simpler programmingmodel for many operations. In one embodiment, the parameters for theimagers are arbitrarily fixed for a target application. In anotherembodiment, the parameters are configured to allow a high degree offlexibility and programmability.

In one embodiment, the camera array is designed as a drop-in replacementfor existing camera image sensors used in cell phones and other mobiledevices. For this purpose, the camera array may be designed to bephysically compatible with conventional image sensors of approximatelythe same resolution although the achieved resolution of the camera arraymay exceed conventional image sensors in many photographic situations.Taking advantage of the increased performance, the camera array inaccordance with embodiments of the invention may include fewer pixels toobtain equal or better quality images compared to conventional imagesensors. Alternatively, the size of the pixels in the imager may bereduced compared to pixels in conventional image sensors while achievingcomparable results.

In order to match the raw pixel count of a conventional image sensorwithout increasing silicon area, the logic overhead for the individualimagers is preferably constrained in the silicon area. In oneembodiment, much of the pixel control logic is a single collection offunctions common to all or most of the imagers with a smaller set offunctions applicable to each imager. In this embodiment, theconventional external interface for the imager may be used because thedata output does not increase significantly for the imagers.

In one embodiment, the camera array including the imagers replaces aconventional image sensor of M megapixels. The camera array includes N×Nimagers, each sensor including pixels of

$\frac{M}{N^{2}}.$Each imager in the camera array also has the same aspect ratio as theconventional image sensor being replaced. Table 1 lists exampleconfigurations of camera arrays according to the present inventionreplacing conventional image sensor.

TABLE 1 Conventional Image Camera array Including Imagers Sensor No. No.of Super- Total Effective Total Horizontal of Vertical Imager ResolutionEffective Mpixels Resolution Mpixels Imagers Imagers Mpixels FactorResolution 8 3.2 8 5 5 0.32 3.2 3.2 8 4 4 0.50 2.6 3.2 8 3 3 0.89 1.93.2 5 2.0 5 5 5 0.20 3.2 2.0 5 4 4 0.31 2.6 2.0 5 3 3 0.56 1.9 2.0 3 1.23 5 5 0.12 3.2 1.2 3 4 4 0.19 2.6 1.2 3 3 3 0.33 1.9 1.2

The Super-Resolution Factors in Table 1 are estimates and the EffectiveResolution values may differ based on the actual Super-Resolutionfactors achieved by processing.

The number of imagers in the camera array may be determined based on,among other factors, (i) resolution, (ii) parallax, (iii) sensitivity,and (iv) dynamic range. A first factor for the size of imager is theresolution. From a resolution point of view, the preferred number of theimagers ranges from 2×2 to 6×6 because an array size of larger than 6×6is likely to destroy frequency information that cannot be recreated bythe super-resolution process. For example, 8 Megapixel resolution with2×2 imager will require each imager to have 2 Megapixels. Similarly, 8Megapixel resolution with a 5×5 array will require each imager to have0.32 Megapixels. In many embodiments, the number of imagers in the arrayis determined based upon the requirements of a specific application.

A second factor that may constrain the number of imagers is the issue ofparallax and occlusion. With respect to an object captured in an image,the portion of the background scene that is occluded from the view ofthe imager can be referred to as the “occlusion set.” When two imagerscapture the object from two different locations, the occlusion set ofeach imager is different. Hence, there may be scene pixels captured byone imager but not the other. To resolve this issue of occlusion, it isdesirable to include a certain minimal set of imagers for a given typeof imager and to distribute the imagers symmetrically around the centralaxis of the camera array.

A third factor that may put a lower bound on the number of imagers isthe issue of sensitivity in low light conditions. To improve low lightsensitivity, imagers for detecting near-IR spectrum may be needed. Thenumber of imagers in the camera array may need to be increased toaccommodate such near-IR imagers.

A fourth factor in determining the size of the imager is dynamic range.To provide dynamic range in the camera array, it is advantageous toprovide several imagers of the same filter type (chroma or luma). Eachimager of the same filter type may then be operated with differentexposures simultaneously. The images captured with different exposuresmay be processed to generate a high dynamic range image.

Based on these factors, the preferred number of imagers is 2×2 to 6×6.4×4 and 5×5 configurations are more preferable than 2×2 and 3×3configurations because the former are likely to provide sufficientnumber of imagers to resolve occlusion issues, increase sensitivity andincrease the dynamic range. In addition, rectangular arrays can also bepreferred. At the same time, the computational load required to recoverresolution from these array sizes will be modest in comparison to thatrequired in the 6×6 array. Arrays larger than 6×6 may, however, be usedto provide additional features such as optical zooming and multispectralimaging. Although only square imagers are described here, as will bediscussed in greater detail later, such imagers may have different x-and y-dimensions.

Another consideration is the number of imagers dedicated to lumasampling. By ensuring that the imagers in the array dedicated to near-IRsampling do not reduce the achieved resolution, the information from thenear-IR images is added to the resolution captured by the luma imagers.For this purpose, at least 50% of the imagers may be used for samplingthe luma and/or near-IR spectra. In one embodiment with 4×4 imagers, 4imagers samples luma, 4 imagers samples near-IR, and the remaining 8imagers sample two chroma (Red and Blue). In another embodiment with 5×5imagers, 9 imagers sample luma, 8 imagers sample near-IR, and theremaining 8 imagers sample two chroma (Red and Blue). Further, theimagers with these filters may be arranged symmetrically within thecamera array to address occlusion due to parallax. In a furtherembodiment with 5×5 imager, 17 imagers sample luma, 4 images sample Red,and 4 imagers sample Blue.

In one embodiment, the imagers in the camera array are spatiallyseparated from each other by a predetermined distance. By increasing thespatial separation, the parallax between the images captured by theimagers may be increased. The increased parallax is advantageous wheremore accurate distance information is important. Separation between twoimagers may also be increased to approximate the separation of a pair ofhuman eyes. By approximating the separation of human eyes, a realisticstereoscopic 3D image may be provided to present the resulting image onan appropriate 3D display device.

In one embodiment, multiple camera arrays are provided at differentlocations on a device to overcome space constraints. One camera arraymay be designed to fit within a restricted space while another cameraarray may be placed in another restricted space of the device. Forexample, if a total of 20 imagers are required but the available spaceallows only a camera array of 1×10 imagers to be provided on either sideof a device, two camera arrays each including 10 imagers may be placedon available space at both sides of the device. Each camera array may befabricated on a substrate and be secured to a motherboard or other partsof a device. In addition, such imagers do not have to be homogenous insize, and may have different x- and y-dimensions. The images collectedfrom multiple camera arrays may be processed to generate images ofdesired resolution and performance.

A design for a single imager may be applied to different camera arrayseach including other types of imagers. Other variables in the cameraarray such as spatial distances, color filters and combination with thesame or other sensors may be modified to produce a camera array withdiffering imaging characteristics. In this way, a diverse mix of cameraarrays may be produced while maintaining the benefits from economies ofscale.

Wafer Level Optics Integration

In one embodiment, the camera array employs wafer level optics (WLO)technology. Although in many embodiments, similar optical channels canbe constructed using any of a variety of techniques including but notlimited to injection molding, glass molding, and/or combinations ofthese techniques with other techniques including WLO techniques. WLOitself is a technology that encompasses a number of processes,including, for example, molding optics (such as arrays of lens modulesand arrays of those lens arrays) on glass wafers, stacking of thosewafers (including wafers having lenses replicated on either side of thesubstrate) with appropriate spacers, at either a wafer or die-level,followed by packaging of the optics directly with the imager into amonolithic integrated module.

The WLO procedure may involve, among other procedures, using adiamond-turned mold to create each polymer lens element on a glasssubstrate. More specifically, the process chain in WLO generallyincludes producing a diamond turned lens master (both on an individualand array level), then producing a negative mold for replication of thatmaster (also called a stamp or tool), and then finally forming a polymerreplica on a glass substrate, which has been structured with appropriatesupporting optical elements, such as, for example, apertures, lightblocking materials, filters, etc.

FIG. 2A is a perspective view of a camera array assembly 200 with waferlevel optics 210 and a sensor array 230, according to one embodiment.The wafer level optics 210 includes a plurality of lens elements 220,each lens element 220 covering one of twenty-five imagers 240 in thesensor array 230. Note that the camera array assembly 200 has an arrayof smaller lens elements occupy much less space compared to a singlelarge lens covering the entire sensor array 230. It should also be notedthat each of the lenses may be of a different type. For example, eachsubstrate level may contain lenses that are diffractive, refractive,Fresnel, or a combination thereof. It should be further noted that inthe context of the camera array, that a lens element 220 may compriseone or multiple separate optical lens elements axially arranged withrespect to each another. Finally, it should be noted that, for most lensmaterials there will be a thermal induced variance in the refractiveindex of the material, which must be corrected to obtain good imagequality. A temperature normalization procedure will be described ingreater detail in the sections to follow. FIG. 2B is a sectional view ofa camera array assembly 250, according to one embodiment. The cameraassembly 250 includes a top lens wafer 262, a bottom lens wafer 268, asubstrate 278 with multiple sensors and associated light sensingelements formed thereon and spacers 258, 264 and 270. The camera arrayassembly 250 is packaged within an encapsulation 254. An optional topspacer 258 may be placed between the encapsulation 254 and the top lenswafer 262; however, it is not essential to the construction of thecamera assembly 250. Optical elements 288 are formed on the top lenswafer 262. Although these optical elements 288 are shown as beingidentical in FIG. 2B, it should be understood that different types,sizes, and shapes of elements may be used. A middle spacer 264 is placedbetween the top lens wafer 262 and a bottom lens wafer 268. Another setof optical elements 286 is formed on the bottom lens wafer 268. A bottomspacer 270 is placed between the bottom lens wafer 268 and the substrate278. Through-silicon vias 274 are also provided to paths fortransmitting signal from the imagers. The top lens wafer 262 may bepartially coated with light blocking materials 284 (see discussionbelow) to block of light. The portions of the top lens wafer 262 notcoated with the blocking materials 284 serve as aperture stops throughwhich light passes to the bottom lens wafer 268 and the light sensingelements. Although only a single aperture stop is shown in theembodiment provided in FIG. 2B, it should be understood that additionalaperture stops may be formed from opaque layers disposed on any and allof the substrate faces in the camera assembly to improve stray lightperformance and reduced optical crosstalk. A fuller discussion ofoptical crosstalk suppression is provided below. In addition, althoughthe above embodiment is shown with spacers 258, 264 and 270 the spacerfunction can also be directly implemented by modifying the lensstructures (or substrates) so that the lenses can be directlyinterconnected. In such an embodiment, the lens height can be extended,and the lens glued directly to the upper substrates thereby eliminatingthe need for spacer layers.

In the embodiment of FIG. 2B, filters 282 are formed on the bottom lenswafer 268. Light blocking materials 280 may also be coated on the bottomlens 268 to function as an optical isolator. A light blocking material280 may also be coated on the substrate 278 to protect the sensorelectronics from incident radiation. Spacers 283 can also be placedbetween the bottom lens wafer 268 and the substrate 278 and between thelens wafers 262, 268. In many embodiments, the spacers 283 are similarto the spacers 264 and 270. In a number of embodiments, each layer ofspacers is implemented using a single plate. Although not illustrated inFIG. 2B, many embodiments of the invention also include spacers betweeneach optical channel located on top of the top lens wafer 262 that aresimilar to, or implemented in single layer with, the spacer 258 shown atthe edge of the lens stack array. As is discussed further below thespacers can be constructed from and/or coated in light blockingmaterials to isolate the optical channels formed by the wafer leveloptics. For the purposes of this application, suitable light blockingmaterials may include any opaque material, such as, for example, a metalmaterial like Ti and Cr, or an oxide of these materials like blackchromium (chrome and chrome oxide), or dark silicon, or a black particlefilled photoresist like a black matrix polymer (PSK2000 from BrewerScience). The bottom surface of the substrate is covered with a backsideredistribution layer (“RDL”) and solder balls 276. In one embodiment,the camera array assembly 250 includes 5×5 array of imagers. The cameraarray 250 has a width W of 7.2 mm, and a length of 8.6 mm. Each imagerin the camera array may have a width S of 1.4 mm. The total height t1 ofthe optical components is approximately 1.26 mm and the total height t2the camera array assembly is less than 2 mm. Other heights t1 and t2 arepossible for different lens designs.

Optical Crosstalk Suppression

As discussed above, the camera array assembly 250 is composed ofmultiple imagers, each of which, as shown in FIGS. 2A and 2B, have acorresponding optical pathway or channel that directs light from thescene through the top lens wafer 262, the middle spacer 264, bottom lenswafer 268, the bottom spacer 270 and onto a plurality of light sensingelements that form a sensor 240 disposed on the substrate 278. It isimportant to final image quality that the light impinging on anyparticular sensor come only from its designated optical pathway orchannel. Optical crosstalk can be considered to occur when light that isincident on the top of one imager is also received by light sensingelements of another imager within the array. Any crosstalk betweenoptical channels from, for example, diffraction and/or scattering oflight from elements within the camera, can introduce artifacts in theimage data. In particular, crosstalk between optical channels means thatan imager will sense the flux from a source on the imager that isinconsistent with the reconstructed position of the image of thatdetector and the position of the image. This results in both a loss ofimage data, and the introduction of overlapping noise that cannot bedistinguished from real image data. Accordingly, all optical channels ofthe camera array should be optically isolated so that a ray of lightfrom one lens or optical channel cannot cross from one optical channelto the other. In one embodiment, shown in FIG. 2C, opaque spacers 281 orvertical opaque walls 282 are disposed between each of the opticalchannels 284. Although opaque spacers do provide a level of opticalcrosstalk suppression, vertical opaque walls are preferable because insuch an embodiment both the space between substrates and the relevantsections of the substrates themselves are rendered non-transparent.

The optical crosstalk suppressing vertical opaque walls may be madeusing any suitable technique that provides for the introduction of anopaque surface or material between the optical channels 284 of thecamera array assembly 286. In one embodiment, the vertical opaque wallsare formed by fully or partially introducing grooves into the lens stack288 of the camera array assembly 286. It is preferable not to cut thegrooves fully through the lens stack to preserve the mechanicalintegrity of the camera array assembly. Such grooves may be introducedby any suitable technique, such as, for example, by dicing into thefront or backside of the lens array stack 286 using a wafer dicer(disk/blade), or by laser cutting, or water-jet cutting. Once thegrooves are formed, they are filled with a light blocking material.Alternatively, the inner side walls of the grooves may be coated with alight blocking material and the remainder of the groove filled withanother material with low shrinkage properties. As discussed above, alight blocking material is any opaque material, such as, for example, ametal material, a metal oxide, dark silicon, or a black particle filledphotoresist like a black matrix polymer.

In another embodiment, shown schematically in FIG. 2D, optical crosstalksuppression is achieved by creating a virtual opaque wall formed by aseries of stacked apertures. In this embodiment, a series of aperturestops are formed on the various substrate levels 290 of the camera arrayassembly 292 by coating the substrates with opaque layers 294 providedwith a narrow opening or aperture 296. If enough of these apertures areformed, it is possible to mimic the optical isolation provided by avertical opaque wall. In such a system, a vertical wall would be themathematical limit of stacking apertures one on top of each other.Preferably, as many apertures as possible, separated from each other bysufficient space, are provided so that such a virtual opaque wall iscreated. For any camera array assembly, the number and placement ofopaque layers needed to form such a virtual vertical opaque wall can bedetermined through a ray tracing analysis.

In a further embodiment shown schematically in FIG. 2E, opticalcrosstalk suppression is achieved using spacers 295 constructed fromopaque materials. In another further embodiment shown schematically inFIG. 2F, optical crosstalk suppression is achieved using spacers 296coated with an opaque coating 297. The embodiments illustrated in FIGS.2E and 2F include stacked apertures 294 similar to the stacked apertures294 illustrated in FIG. 2D. In several embodiments, optical crosstalksuppression is achieved without using stacked apertures. In manyembodiments, any of a variety of light blocking materials can be used inthe construction or coating of spacers to achieve optical isolation.

Lens Properties

FIGS. 3A and 3B are diagrams illustrating changes in the height t of alens element pursuant to changes in dimensions in an x-y plane. A lenselement 320 in FIG. 3B is scaled by 1/n compared to a lens element 310in FIG. 3A. Note that during scaling it is important to keep the same F#so image properties don't change. As the diameter L/n of the lenselement 320 is smaller than the diameter L by a factor of n, the heightt/n of the lens element 320 is also smaller than the height t of thelens element 310 by a factor of n. Hence, by using an array of smallerlens elements, the height of the camera array assembly can be reducedsignificantly. The reduced height of the camera array assembly may beused to design less aggressive lenses having better optical propertiessuch as improved chief ray angle, reduced distortion, and improved coloraberration.

FIG. 3C illustrates improving a chief ray angle (CRA) by reducing thethickness of the camera array assembly. CRA1 is the chief ray angle fora single lens covering an entire camera array. Although the chief rayangle can be reduced by increasing the distance between the camera arrayand the lens, the thickness constraints impose constraints on increasingthe distance. Hence, the CRA1 for camera array having a single lenselement is large, resulting in reduced optical performance. CRA2 is thechief ray angle for an imager in the camera array that is scaled inthickness as well as other dimensions. The CRA2 remains the same as theCRA1 of the conventional camera array and results in no improvement inthe chief ray angle. By modifying the distance between the imager andthe lens element as illustrated in FIG. 3C, however, the chief ray angleCRA3 in the camera array assembly may be reduced compared to CRA1 orCRA2, resulting in better optical performance. As described above, thecamera arrays according to the present invention has reduced thicknessrequirements, and therefore, the distance of the lens element and thecamera array may be increased to improve the chief ray angle. Thisrelaxed CRA, in turn, results in a lower F# and improved ModulationTransfer Function (MTF).

Specifically, one of the issues raised in camera design is how tocorrect for field curvature. An image projected through a lens is notplanar, but has an inherently curved surface. One way to correct thisfield curvature is to position a thick negative lens element 312 closeto or directly on the imager surface 314. The negative lens elementplanarizes the various angled beams of light 316 from the image, therebyaddressing the field curvature problem. Such field flattened imagesprovide superior image performance, allow for the manufacture of arraycameras with relaxed TTL requirements, and deliver very homogeneous MTF.However, one problem with this approach is that this field flatteningapproach intrinsically requires a high CRA. This makes the techniqueunsuitable for most cameras; however, camera arrays in accordance withembodiments of the invention allow for the use of backside imaging(BSI). Positioning the image sensor behind the substrate relaxes the CRAangle requirement, thereby allowing for the use of the negative lenselement field flattening approach shown in FIG. 3D.

Another advantage of the array camera relates to chromatic aberrations.Specifically, in a conventional polychromatic lens, the lens has to becorrected for chromatic aberrations, because the focal length throughthe lens is different for different wavelengths of light. As a result,it is necessary to compromise the performance of the lens for some ofthe color wavelengths to get acceptable overall color performance. Bymaking each optical channel narrow spectral band, color aberration isreduced and/or prevented, and each lens may be optimized to a specificcolor wavelength. For example, an imager receiving visible or near-IRspectrum may have a lens element specifically optimized for thisspectral band of light. For imagers detecting other light spectrum, thelens element may be constructed with different properties, such as radiiof curvature, so that a constant focal length across all wavelengths oflight is achieved so that, in turn, the focal plane is the same fordifferent spectral bands of light. The matching of the focal planeacross different wavelengths of light increases the sharpness of imagecaptured at the imager and reduces longitudinal chromatic aberration.Because each lens element may be designed to direct a narrow band oflight, the concomitant lack of color aberration means that the lenselements can be subject to less rigorous design constraints, yet producebetter or equivalent performance compared to a conventional lens elementcovering a wide light spectrum. In particular, there is no need toundertake costly aberration balancing correction. What is more, simplelenses generally have better MTF and lower F# (higher sensitivity). Itshould be noted that although the lenses used in these array camerashave much smaller color aberrations when compared to conventionalpolychromatic lenses, each lens is still designed to focus a certainwavelength-bandwidth. Accordingly, in one embodiment each of these“monochromatic” lenses can be optimally color corrected by usingcombinations of high and low Abbe number materials (different opticaldispersions).

Light of different wavelengths having different focal lengths(longitudinal color aberration) is not the only type of aberration thatoccurs in polychromatic optical systems. The refractive index of a lensis dependent on the wavelength of light passing through the lens. As aresult, a lens will impart different magnification to colors ofdifferent wavelengths. For example, the red wavelength band might have aslightly smaller magnification than green, and green may in turn have aslightly smaller magnification than blue. If the images obtained fromthese different wavelengths of light are then overlaid withoutcorrection, the image will lose resolution because the different colorswill not overlap correctly. Based on the properties of the material, thedifferential lateral distortions of the color magnification can bedetermined and then corrected. Correction can be accomplished byrestricting the profiles of the lenses so that each color has the samemagnification, but this reduces the possible degrees of freedomavailable for lens manufacture, and reduces the ability to optimize MTF.Accordingly, lateral distortion can be permitted optically, and thencorrected after imaging computationally. The electronic correction ofthe lateral color of the lens can actually provide improvements tosystem performance above and beyond simply correcting for the originaldistortion, because such correction directly improves the resolution ofthe system in terms of polychromatic MTF. In particular, lateral coloraberrations in a lens can be seen as a color dependent distortion of thelens. By mapping all differently distorted single color images of anobject back to the same rectangle, perfect overlap can be achieved inthe full color image resulting in the polychromatic MTF being the sameas the monochromatic one (not only due to the individual color channelcolor-blur correction, but also as a result of the exact superpositionof the different colors).

Yet another advantage to using many lenses, each optimized for use witha narrow band of light, is that the there is no restriction on the typeof lens that may be used. In particular, the array camera allows for theuse of diffractive, refractive, Fresnel lenses, or combinations of thesetypes of lenses. Diffractive lenses are attractive because they allowfor the creation of complex wavefronts with an essentially flat opticalelement, and they are also relatively simple to manufacture. Inconventional cameras it is not possible to use diffractive lensesbecause having a single imager means that the lens must be able toefficiently transmit a wide spectrum of light, and while diffractivelenses are very efficient at transmitting narrow wavelength bands oflight, there is a steep drop-off in performance for wavelengths of lightoutside of this optimized range. Because each array of the currentcamera may be focused on a narrow wavelength of light, the narrowoptimized wavelength band of these diffractive lenses is not a limitingfactor.

Other advantages of smaller lens elements include, among others, reducedcost, reduced amount of materials, and the reduction in themanufacturing steps. By providing n² lenses that are 1/n the size in xand y dimension (and thus 1/n thickness), the wafer size for producingthe lens element may also be reduced. This reduces the cost and theamount of materials considerably. Further, the number of lens substratesis reduced, which results in a reduced number of manufacturing steps andreduced attendant yield costs. The placement accuracy required toregister the lens array to the imagers is typically no more stringentthan in the case of a conventional imager because the pixel size for thecamera array according to the present invention may be substantially thesame as a conventional image sensor. In addition, monochromaticaberrations scale with lens diameter. Because array cameras are able touse smaller lenses, any aberrations that exist are smaller so it ispossible to use lenses with simpler profiles. This results in a systemthat is simultaneously better and less costly to fabricate. Smallersized lenses also have a lower volume, which results in lower sag orshrinkage during manufacture. Shrinkage is bad for replication becauseit deforms the desired lens profile, and results in the need for thefabricator to precompensate for the predicted level of sag so that thefinal lens shape will be correct. This precompensation is difficult tocontrol. With lower sag/shrinkage it is not necessary to have thesetight fabrication controls, again lowering the overall cost of themanufacture of the lenses.

In one embodiment, the WLO fabrication process includes: (i)incorporating lens element stops by plating the lens element stops ontothe substrate before lens molding, and (ii) etching holes in thesubstrate and performing two-sided molding of lenses through thesubstrate. The etching of holes in the substrate is advantageous becauseindex mismatch is not caused between plastic and substrate. In this way,light absorbing substrate that forms natural stops for all lens elements(similar to painting lens edges black) may be used.

In one embodiment, filters are part of the imager. In anotherembodiment, filters are part of a WLO subsystem. In an embodimentincluding a filter, it is preferred to dispose the filter (whether CFA,IR and/or VIS) into or close to the aperture stop surface and not at theimager sensor surface, because when positioned at a distance from theimager sensor small defects in those filter layers are averaged out overall entrance pupil positions, and are therefore less visible.

Imaging System and Processing Pipeline

FIG. 4 is a functional block diagram illustrating an imaging system 400,according to one embodiment. The imaging system 400 may include, amongother components, the camera array 410, an image processing pipelinemodule 420 and a controller 440. The camera array 410 includes two ormore imagers, as described above in detail with reference to FIGS. 1 and2. Images 412 are captured by the two or more imagers in the cameraarray 410.

The controller 440 is hardware, software, firmware or a combinationthereof for controlling various operation parameters of the camera array410. The controller 440 receives inputs 446 from a user or otherexternal components and sends operation signals 442 to control thecamera array 410. The controller 440 may also send information 444 tothe image processing pipeline module 420 to assist processing of theimages 412.

The image processing pipeline module 420 is hardware, firmware, softwareor a combination for processing the images received from the cameraarray 410. The image processing pipeline module 420 processes multipleimages 412, for example, as described below in detail with reference toFIG. 5. The processed image 422 is then sent for display, storage,transmittal or further processing.

FIG. 5 is a functional block diagram illustrating the image processingpipeline module 420, according to one embodiment. The image processingpipeline module 420 may include, among other components, an upstreampipeline processing module 510, an image pixel correlation module 514, aparallax confirmation and measurement module 518, a parallaxcompensation module 522, a super-resolution module 526, an addressconversion module 530, an address and phase offset calibration module554, and a downstream color processing module 564.

The address and phase offset calibration module 554 is a storage devicefor storing calibration data produced during camera arraycharacterization in the manufacturing process or a subsequentrecalibration process. In several embodiments, the calibration data canindicate mapping between the addresses of physical pixels 572 in theimagers and the logical addresses 546, 548 of an image. In otherembodiments, a variety of calibration data appropriate to a specificapplication can be utilized in the address and phase offset calibrationmodule.

The address conversion module 530 performs normalization based on thecalibration data stored in the address and phase offset calibrationmodule 554. Specifically, the address conversion module 530 converts“physical” addresses of the individual pixels in the image to “logical”addresses 548 of the individual pixels in the imagers or vice versa. Inorder for super-resolution processing to produce an image of enhancedresolution, the phase difference between corresponding pixels in theindividual imagers needs to be resolved. The super-resolution processmay assume that for each pixel in the resulting image the set of inputpixels from each of the imager is consistently mapped and that the phaseoffset of the image captured by each imager is already known withrespect to the position of the pixel in the resulting image.Alternatively, the pixel offsets can be estimated prior to thesuperresolution process. The address conversion module 530 resolves suchphase differences by converting the physical addresses in the images 412into logical addresses 548 of the resulting image for subsequentprocessing.

The images 412 captured by the imagers 540 are provided to the upstreampipeline processing module 510. The upstream pipe processing module 510may perform one or more of normalization of the color planes, BlackLevel calculation and adjustments, fixed noise compensation, optical PSF(point spread function) deconvolution, noise reduction, lateral colorcorrection and crosstalk reduction.

In one embodiment, the upstream pipeline processing module also performstemperature normalization. Temperature normalization corrects forchanges in the refractive index of the optical components through whichthe imagers receive light that result from changes in the temperature ofthe camera during use. In several embodiments, the temperaturenormalization process involves determining the temperature of the cameraarray by measuring the dark current of one or an average of a number ofthe camera array's imagers. Using this measurement, a refractive indexnormalization is performed by picking the correct point spread functionfrom temperature calibration data. Different point spread functions maybe obtained during a temperature dependent refractive indexcharacterization of the camera during manufacture, and stored in theimaging system for use in the temperature normalization process.

After the image is processed by the upstream pipeline processing module510, an image pixel correlation module 514 performs calculation toaccount for parallax that becomes more apparent as objects beingcaptured approach the camera array. Specifically, the image pixelcorrelation module 514 aligns portions of images captured by differentimagers to compensate for the parallax. In one embodiment, the imagepixel correlation module 514 compares the difference between the averagevalues of neighboring pixels with a threshold and flags the potentialpresence of parallax when the difference exceeds the threshold. Thethreshold may change dynamically as a function of the operatingconditions of the camera array. Further, the neighborhood calculationsmay also be adaptive and reflect the particular operating conditions ofthe selected imagers.

The image is then processed by the parallax confirmation and measurementmodule 518 to detect and meter the parallax. In one embodiment, parallaxdetection is accomplished by a running pixel correlation monitor. Thisoperation takes place in logical pixel space across the imagers withsimilar integration time conditions. When the scene is at practicalinfinity, the data from the imagers is highly correlated and subjectonly to noise-based variations. When an object is close enough to thecamera, however, a parallax effect is introduced that changes thecorrelation between the imagers. Due to the spatial layout of theimagers, the nature of the parallax-induced change is consistent acrossall imagers. Within the limits of the measurement accuracy, thecorrelation difference between any pair of imagers dictates thedifference between any other pair of imagers and the differences acrossthe other imagers. This redundancy of information enables highlyaccurate parallax confirmation and measurement by performing the same orsimilar calculations on other pairs of imagers. If parallax is presentin the other pairs, the parallax should occur at roughly the samephysical location of the scene taking into account the positions of theimagers. The measurement of the parallax may be accomplished at the sametime by keeping track of the various pair-wise measurements andcalculating an “actual” parallax difference as a least squares (orsimilar statistic) fit to the sample data. Other methods for detectingthe parallax may include detecting and tracking vertical and horizontalhigh-frequency image elements from frame-to-frame.

The parallax compensation module 522 processes images including objectsclose enough to the camera array to induce parallax differences largerthan the accuracy of the phase offset information required by superresolution process. The parallax compensation module 522 uses thescan-line based parallax information generated in the parallax detectionand measurement module 518 to further adjust mapping between physicalpixel addresses and logical pixel addresses before the super-resolutionprocess. There are two cases that occur during this processing. In amore common case, addressing and offsetting adjustment are required whenthe input pixels have shifted positions relative to theimage-wise-corresponding pixels in other imagers. In this case, nofurther processing with respect to parallax is required beforeperforming super-resolution. In a less common case, a pixel or group ofpixels are shifted in such a way that exposes the occlusion set. In thiscase, the parallax compensation process generates tagged pixel dataindicating that the pixels of the occlusion set should not be consideredin the super-resolution process.

After the parallax change has been accurately determined for aparticular imager, the parallax information 524 is sent to the addressconversion module 530. The address conversion module 530 uses theparallax information 524 along with the calibration data 558 from theaddress and phase offset calibration module 554 to determine theappropriate X and Y offsets to be applied to logical pixel addresscalculations. The address conversion module 530 also determines theassociated sub-pixel offset for a particular imager pixel with respectto pixels in the resulting image 428 produced by the super-resolutionprocess. The address conversion module 530 takes into account theparallax information 524 and provides logical addresses 546 accountingfor the parallax.

After performing the parallax compensation, the image is processed bythe super-resolution module 526 to obtain a high resolution synthesizedimage 422 from low resolution images, as described below in detail. Thesynthesized image 422 may then be fed to the downstream color processingmodule 564 to perform one or more of the following operations: focusrecover, white balance, color correction, gamma correction, RGB to YUVcorrection, edge-aware sharpening, contrast enhancement and compression.

The image processing pipeline module 420 may include components foradditional processing of the image. For example, the image processingpipeline module 420 may include a correction module for correctingabnormalities in images caused by a single pixel defect or a cluster ofpixel defects. The correction module may be embodied on the same chip asthe camera array, as a component separate from the camera array or as apart of the super-resolution module 526.

Super-Resolution Processing

In one embodiment, the super-resolution module 526 generates a higherresolution synthesized image by processing low resolution imagescaptured by the imagers 540. The overall image quality of thesynthesized image is higher than images captured from any one of theimagers individually. In other words, the individual imagers operatesynergistically, each contributing to higher quality images using theirability to capture a narrow part of the spectrum without sub-sampling.The image formation associated with the super-resolution techniques maybe expressed as follows:y _(k) =W _(k) ·x+n _(k) ,∀k=1 . . . p  equation(2)where W_(k) represents the contribution of the HR scene (x) (viablurring, motion, and sub-sampling) to each of the LR images (y_(k))captured on each of the k imagers and n_(k) is the noise contribution.Imager Configurations

FIGS. 6A through 6F illustrate various configurations of imagers forobtaining a high resolution image through a super-resolution process,according to embodiments of the present invention. In FIGS. 6A through6F, “R” represents an imager having a red filter, “G” represents aimager having a green filter, “B” represents an imager having a bluefilter, “P” represents a polychromatic imager having sensitivity acrossthe entire visible spectra and near-IR spectrum, and “I” represents animager having a near-IR filter. The polychromatic imager may sampleimage from all parts of the visible spectra and the near-IR region(i.e., from 650 nm to 800 nm). In the embodiment of FIG. 6A, the centercolumns and rows of the imagers include polychromatic imagers. Theremaining areas of the camera array are filled with imagers having greenfilters, blue filters, and red filters. The embodiment of FIG. 6A doesnot include any imagers for detecting near-IR spectrum alone.

The embodiment of FIG. 6B has a configuration similar to conventionalBayer filter mapping. This embodiment does not include any polychromaticimagers or near-IR imagers. As described above in detail with referenceto FIG. 1, the embodiment of FIG. 6B is different from conventionalBayer filter configuration in that each color filter is mapped to eachimager instead of being mapped to an individual pixel.

FIG. 6C illustrates an embodiment where the polychromatic imagers form asymmetric checkerboard pattern. FIG. 6D illustrates an embodiment wherefour near-IR imagers are provided. FIG. 6E illustrates an embodimentwith irregular mapping of imagers. FIG. 6F illustrates an embodimentwhere a 5×5 sensor array is organized into 17 imagers having greenfilters, four imagers having red filters, and four imagers having bluefilters. The sensors are distributed symmetrically around the centralaxis of the imaging array. As is discussed further below, distributingthe imagers in this way prevents pixels that can be imaged by a sensorfrom being occluded from sensors capturing other wavelengths of light.The embodiments of FIGS. 6A through 6F are merely illustrative andvarious other layouts of imagers can also be used.

The use of polychromatic imagers and near-IR imagers is advantageousbecause these sensors may capture high quality images in low lightingconditions. The images captured by the polychromatic imager or thenear-IR imager are used to denoise the images obtained from regularcolor imagers. However, as discussed above, these polychromatic lensesrequire that an associated color correction technique be used to addresscolor aberrations inherent in a single lens trying to capture allwavelengths of light and deliver it to the same focal plane. Anyconventional color correction technique may be utilized with theproposed array cameras.

Imager Layout

The premise of increasing resolution by aggregating multiple lowresolution images relies upon the different low resolution imagesrepresenting slightly different viewpoints of the same scene. If the LRimages are all shifted by integer units of a pixel, then each imagecontains essentially the same information. Therefore, there is no newinformation in the LR images that can be used to create a HR image. Incamera arrays according to embodiments of the invention, the layout ofthe imagers in the array may be preset and controlled so that eachimager in a row or a column captures an image that is shifted a fixedsub-pixel distance relative to the images captured by its neighboringimagers. Ideally, the images captured by each imager are spatiallyoffset from the other imagers in such a way as to provide uniformsampling of the scene or the light field and the uniformity of samplingis such that the LR images captured by each of the imagers yieldsnon-redundant information about the sampled scene (light field). Suchnon-redundant information about the scene can be utilized by subsequentsignal processing processes to synthesize a single HR image.

A sub-pixel shift between the images captured by two imagers is not,however, sufficient to ensure uniformity of sampling. The uniformity ofsampling or sampling diversity of two imagers is a function of objectdistance. The sampled space by pixels of a pair of imagers isillustrated in FIG. 6G. A first set of rays (610) map to pixels ofimager A, while a second set of rays (620) map to pixels of imager B.Conceptually, two adjacent rays from a given imager define the part ofthe object space that is sampled by a specific pixel in that imager. Atdistance z1 from the camera plane, there is sufficient samplingdiversity since the rays of the pixels of imager A are partially offsetfrom the rays of the pixels of imager B. As the distance decreases thereare specific distances (z2, z3, z4) where there is no sampling diversitybetween imager A and imager B. The lack of sampling diversity betweenthe two imagers quite simply implies that there is no additionalinformation in the scene captured by imager B as compared to thatcaptured by imager A. As is discussed further below, an increased numberof imagers in an array camera can mitigate the impact of objectdistances at which pair of imagers' sample space fully overlap. When apair of imagers lack sampling diversity, the other imagers in the arrayprovide the necessary sampling diversity to achieve resolutionenhancement. Consequently, the ability of an imager system utilizing a2×2 array of imagers to achieve superresolution is typically morelimited than camera systems in accordance with embodiments of theinvention that use a larger camera array.

Referring back to the camera array structures illustrated in FIGS. 2A-2Dthe wafer level optics includes a plurality of lens elements, where eachlens element covers one of the sensors in the array. The physical layoutof pixels in a single imager of a camera array in accordance with anembodiment of the invention is illustrated in FIG. 6H. The imager is anarray of pixels 650 overlaid with color filters 652 and microlenses 654.The microlenses that sit on top of the color filters are used to focuslight on the active area of each underlying pixel. The microlenses canbe thought of as sampling the continuous light field in object spacesampled by the main lens. Whereas the main lens samples the sceneradiance light field, the micro-lenses sample the sensor irradiancelight field.

The main lens associated with each imager maps the points in the objectspace to points in the image space such at that the mapping is bijective(onto-to-one and onto). Each microlens samples a finite extent of thesensor irradiance light field. The sensor irradiance light field iscontinuous and is the result of a bijective mapping from the objectspace. Thus, the microlens sampling of a finite extent of the sensorirradiance light field is also a sampling of a corresponding finiteextent of the scene radiance light field in object space.

Moving the microlens by a small amount δ laterally along the plain ofthe imager pixels changes the sampled object space at a certain distancez_(k) by a correspondingly appropriate factor δ. With an n×n (n>2) arraycamera, we can choose a baseline microlens shift can be determined bythe main lens profile (for example, the chief ray angle) for a baselineimager. For each of the other imagers that sample the same wavelength asthe baseline imager, the microlenses of each of the pixels in the imagerare shifted by a sub-pixel amount to sample a different part of thescene radiance light field. Thus for a set of imagers arranged in an n×ngrid, the sub-pixel shift for a imager that images the same wavelengthsas the baseline imager (1,1) at a grid location (i,j) (1≤i,j≤n) isgoverned by (δ_(x), δ_(y)) where,

${\frac{( {i - 1} )}{n} \times \mspace{14mu}{pixelsize}}\mspace{14mu} < \delta_{x} \leq {\frac{i}{n} \times \mspace{14mu}{pixelsize}}$${\frac{( {j - 1} )}{n} \times \mspace{14mu}{pixelsize}}\mspace{14mu} < \delta_{y} \leq {\frac{j}{n} \times \mspace{14mu}{pixelsize}}$

Many camera arrays in accordance with embodiments of the inventioninclude significantly more Green imagers than Red and Blue imagers. Forexample, the array camera illustrated in FIG. 6F includes 17 Greenimagers, 4 Red imagers, and 4 Blue imagers. For the purpose ofcalculating the sub-pixel shifts, the Green imagers can be treated as ann×n grid. Whereas the Red imagers and the Blue imagers can each betreated as a 2×2 grid for the purpose of calculating the sub-pixelshifts.

The sub-pixel shifts discussed above are determined relative to abaseline imager located at the corner of the grid, many embodiments ofthe invention utilize radial sub-pixel shifts from a baseline imagerlocated at the center of the sensor array. In several embodiments, theradial sub-pixel shifts are chosen so the sub-pixel shifts are evenlydistributed to enable the greatest sampling diversity.

The constraints on microlens sub-pixel shifts defined above achieve thehighest increases in diversity and can enable the greatest increases inresolution through superresolution processing. Sub-pixel shifts that donot satisfy the constraints, but still provide an increase in samplingdiversity can also be used to enable some increase in resolution throughsuperresolution processing. Therefore, embodiments of the invention arenot limited to microlens shifts that result in the greatest increases indiversity and in many instances utilize a variety of different microlensshift configurations that provide at least some increase in samplingdiversity and that are satisfactory for the requirements of a specificapplication.

Symmetry of Imager Placement in Camera Array

An issue of separating the spectral sensing elements into differentimagers is parallax caused by the physical separation of the imagers. Byensuring that the imagers are symmetrically placed, at least two imagerscan capture the pixels around the edge of a foreground object. In thisway, the pixels around the edge of a foreground object may be aggregatedto increase resolution as well as avoiding any occlusions. In theabsence of a symmetrical distribution, a pixel around the edge of aforeground object that is visible to a first imager, for example a Redimager, may be occluded to a second imager that captures differentwavelengths, for example a blue imager. Accordingly, color informationfor the pixel cannot be accurately reconstructed. By symmetricallydistributing the sensors, the likelihood that a foreground object willocclude pixels is significantly reduced.

Pixel occlusion caused by an asymmetric distribution of Red and Blueimagers in a simple array is illustrated in FIG. 6I. A pair of Redimagers 672 is located on the left hand side of the camera array 670 anda pair of Blue imagers 674 is located on the right hand side of thecamera array. A foreground object 676 is present and the Red imagers 672are capable of imaging regions beyond the foreground object on the lefthand side of the foreground object. However, the foreground objectoccludes the Red imagers from imaging these regions. Therefore, thearray camera is incapable of reconstructing color information for theseregions.

An array that includes a symmetric distribution of Red and Blue imagersin accordance with an embodiment of the invention is illustrated in FIG.6J. The camera array 780 includes a pair of Red imagers 782symmetrically distributed around the central axis of the camera arrayand a pair of Blue imagers 784 symmetrically distributed around thecentral axis of the camera array. Due to the even distribution, a Redimager and a Blue imager are both able to image beyond the foregroundobject 786 on the left hand side of the foreground object and a Redimager and a Blue imager are both able to image beyond the foregroundobject on the right hand side of the foreground object.

The symmetrical arrangement of the simple embodiment illustrated in FIG.6J can be generalized to array cameras including Red, Green, Blueimagers and/or additional polychromatic or near-IR cameras. Bydistributing each of the different types of imagers symmetrically aroundthe central axis of the camera array, the effects of parallax introducedby foreground objects can be significantly reduced and color artifactsthat would otherwise be introduced, avoided.

The effects of parallax on the sampling of color can also be reduced byusing parallax information in polychromatic imagers to improve theaccuracy of the sampling of color from the color filtered imagers.

Use of Near-IR Imagers to Obtain Improved High Resolution Images

In one embodiment, near-IR imagers are used to determine relativeluminance differences compared to a visible spectra imager. Objects havediffering material reflectivity results in differences in the imagescaptured by the visible spectra and the near-IR spectra. At low lightingconditions, the near-IR imager exhibits a higher signal to noise ratios.Therefore, the signals from the near-IR sensor may be used to enhancethe luminance image. The transferring of details from the near-IR imageto the luminance image may be performed before aggregating spectralimages from different imagers through the super-resolution process. Inthis way, edge information about the scene may be improved to constructedge-preserving images that can be used effectively in thesuper-resolution process. The advantage of using near-IR imagers isapparent from equation (2) where any improvement in the estimate for thenoise (i.e., n) leads to a better estimate of the original HR scene (x).

Generation of High Resolution Images

FIG. 7 is a flowchart illustrating a process of generating an HR imagefrom LR images captured by a plurality of imagers, according to oneembodiment. First, luma images, near-IR images and chroma images arecaptured 710 by imagers in the camera array. Then normalization isperformed 714 on the captured images. The images can be normalized in avariety of ways including but not limited to normalizing the colorplanes of the images, performing temperature compensation, and mappingphysical addresses of the imagers to logical addresses in the enhancedimage. In other embodiments, a variety of normalization processappropriate to the specific imagers and imaging applications. Parallaxcompensation is then performed 720 to resolve any differences in thefield-of-views of the imagers due to spatial separations between theimagers. Super-resolution processing is then performed 724 to obtainsuper-resolved luma images, super-resolved near-IR images, andsuper-resolved chroma images.

Then it is determined 728 if the lighting condition is better than apreset parameter. If the lighting condition is better than theparameter, the process proceeds to normalize 730 a super-resolvednear-IR image with respect to a super-resolved luma image. A focusrecovery is then performed 742. In one embodiment, the focus recovery isperformed 742 using PSF (point spread function) deblurring per eachcolor channel. Then the super-resolution is processed 746 based onnear-IR images and the luma images. A synthesized image is thenconstructed 750.

If it is determined 728 that the lighting condition is not better thanthe preset parameter, the super-resolved near-IR images and luma imagesare aligned 734. Then the super-resolved luma images are denoised 738using the near-IR super-resolved images. Then the process proceeds toperforming focus recovery 742 and repeats the same process as when thelighting condition is better than the preset parameter. Then the processterminates.

Normalization of Color Planes

The relative response of each of the Red, Green, Blue imagers across theimaging planes varies. The variance can be the result of many factorsincluding the optical alignment of the lens and asymmetrical sensorlight path geometry. For a given lens and imager, the variance can becompensated for by calibration and normalization. Without compensation,the variance can give rise to artifacts such as color shading.

A process for normalizing a imager with respect to a baseline imager,which is typically a Green imager located in the center of the cameraarray, in accordance with an embodiment of the invention is discussedbelow with reference to the normalization of a Red imager with respectto a baseline Green imager. A similar process can be used to normalizeBlue imagers with respect to a baseline Green imager. In manyembodiments, the process is applied to normalize each Red and Blueimager in a camera array.

A normalization surface can be calibrated by first capturing a scenewith flat reflectance, and calculating a color ratio surface to serve asthe basis for normalization. An ideal normalization surface is uniformand can be described as:Color Ratio G/R=G(i,j)/R(i,j)=K=G _(center) /R _(center)where (i,j) describe the pixel position, K is a constant, andG_(center), and R_(center), describe the pixel value at the centerposition.

The output pixel values of the calibration scene contain the ideal pixelvalues plus noise plus black level offset, and can be described asfollows:SR(i,j)=R(i,j)+NoiseR(i,j)+black offsetSG(i,j)=G(i,j)+NoiseG(i,j)+black offsetwhere SR, and SG are the output pixel values from each imager.

A process for calibrating the sensor in accordance with an embodiment ofthe invention is illustrated in FIG. 7A. The process 760 includesremoving (762) the black level offset from the sensor pixel values, andlow pass filtering (764) the image planes to reduce noise. Thenormalization plane is calculated (766) and several embodiments arecalculated as follows:Norm R=G(i,j)/(R(i,j)×(G _(center) /R _(center)))where G_(center), and R_(center), are the pixel values at the centerposition.

Following the calculation of the normalization plane, an averagingfilter can be applied (768) and the values of the Norm R plane arestored (770).

The cost of carrying all of the normalization data for each of thesensors in a sensor array can be quite high. Therefore, many embodimentsscan the Norm R plane using a space filling curve to form a onedimensional array. The resultant one dimensional array can be modeled ina variety of different ways including being modeled as a polynomial withsuitable order. In several embodiments, the polynomials of the fittedpolynomial are stored (810) as parameters that are used duringcalibration to reconstruct the two dimensional normalization plane. Theconstruction of a space filling curve in accordance with severalembodiments of the invention is discussed further below.

In several embodiments, a space filling curve is used to form a onedimensional array describing a normalization plane. A space fillingcurve, which is constructed using a spiral scan, is illustrated in FIG.7B. A space filling curve 780 can be constructed by starting at thecenter of the normalization plane 781 and traversing a four sided squareoutwards. Each side of the square expands by two pixels compared to theprevious square such that every pixel will be traversed exactly once. Inthe illustrated embodiment, each position 782 which is marked with an‘X’ corresponds to a valid pixel position. The imager may not have asquare geometry, so the scan path may traverse empty space (indicated asdashed lines). For each position traversed, if it is a valid pixelposition, a new data entry is added to the one dimensional data array.Otherwise, the traversing continues without adding a new value to thedata array. In many embodiments, the one dimensional data array can beefficiently approximated using a 6^(th) order polynomial that can berepresented using the seven coefficients of the polynomial. Given thatcalibration data is typically required for each Red and Blue imager,expressing the normalization planes as coefficients of a polynomialrepresents a significant reduction in storage requirements. In manyembodiments, higher or lower order polynomials, other functions, and/orother compressed representations are utilized to represent thenormalization plane in accordance with the requirements of a specificapplication.

The data value along each side exhibits a fixed geometric relationship.The optical path to the focal point of the lens is shorter for the cellsnear the center line. The base sensitivity can be thought of as a onedimensional center cut of the calibration surface and approximated by alow order polynomial. The sensitivity polynomial can be either stored asa machine constant (i.e., common to all devices with the same design),or stored along with the scan polynomial to provide additionalflexibility. Accordingly, many embodiments of the invention adjust thepixel value based upon the distance factor as follows. For each sidescan, one of the coordinates will be a constant, i.e., constant ‘y’ forhorizontal scan and constant ‘x’ for vertical scan. For each pixel inthe side scan, the sensitivity factor is adjusted towards the constant‘x’ or ‘y’ distance.

By way of example, for a horizontal scan the base value can be found byevaluating the sensitivity polynomial based on the distance ‘y’ from thecenter. In many embodiments, a suitable polynomial is a fourth orderpolynomial. Although other polynomials and/or other functions can beutilized in accordance with the requirements of a specific application.For each pixel in the scan path, the distance from the surface origin isused to find the corresponding sensitivity from the polynomial in thesame manner. The pixel value is multiplied by an adjustment factor andthen stored in the scanned data array. This adjustment factor iscalculated by dividing the base value with the current sensitivityvalue. For the vertical scan a similar method can be applied. Althoughthe example uses a polynomial based sensitivity adjustment, othersensitivity functions and/or adjustments can be utilized depending uponthe requirements of a specific application in accordance with variousembodiments of the invention.

Once calibration data has been obtained for an imager, the calibrationdata can be used in the normalization of pixel information captured bythe imager. The process typically involves retrieving the storedcalibration data, removing the black offset from the captured image andmultiplying the resultant values with the normalization plane. When thenormalization plane is expressed as a polynomial in the manner outlinedabove, the polynomial is used to generate a one-dimensional array and aninverse scan of the one-dimensional array is used to form the twodimensional normalization plane. Where a sensitivity adjustment wasapplied during calibration, an adjustment factor is calculated that isthe reciprocal of the adjustment factor applied during the calibrationscan and the adjustment factor is applied to the values in the onedimensional array during the inverse scan. When other space fillingcurves, representations of the resulting one dimensional data array,and/or sensitivity adjustments are performed during the calibrationprocess, the normalization process is adjusted accordingly.

As can be readily appreciated, calibration and normalization processesin accordance with embodiments of the invention can be applied to eachof the Red and Blue imagers in the camera array. In many embodiments, aGreen imager located in the center of the camera array is used whenperforming the calibration. In other embodiments, a different Greenimager and/or multiple Green imagers can be utilized in the calibrationof the Red and Blue imagers in the camera array.

Image Fusion of Color Images with Near-IR Images

The spectral response of CMOS imagers is typically very good in thenear-IR regions covering 650 nm to 800 nm and reasonably good between800 nm and 1000 nm. Although near-IR images having no chromainformation, information in this spectral region is useful in lowlighting conditions because the near-IR images are relatively free ofnoise. Hence, the near-IR images may be used to denoise color imagesunder the low lighting conditions.

In one embodiment, an image from a near-IR imager is fused with anotherimage from a visible light imager. Before proceeding with the fusion, aregistration is performed between the near-IR image and the visiblelight image to resolve differences in viewpoints. The registrationprocess may be performed in an offline, one-time, processing step. Afterthe registration is performed, the luminance information on the near-IRimage is interpolated to grid points that correspond to each grid pointon the visible light image.

After the pixel correspondence between the near-IR image and the visiblelight image is established, a denoising and detail transfer process maybe performed. The denoising process allows transfer of signalinformation from the near-IR image to the visible light image to improvethe overall SNR of the fusion image. The detail transfer ensures thatedges in the near-IR image and the visible light image are preserved andaccentuated to improve the overall visibility of objects in the fusedimage.

In one embodiment, a near-IR flash may serve as a near-IR light sourceduring capturing of an image by the near-IR imagers. Using the near-IRflash is advantageous, among other reasons, because (i) the harshlighting on objects of interest may be prevented, (ii) ambient color ofthe object may be preserved, and (iii) red-eye effect may be prevented.

In one embodiment, a visible light filter that allows only near-IR raysto pass through is used to further optimize the optics for near-IRimaging. The visible light filter improves the near-IR optics transferfunction because the light filter results in sharper details in thenear-IR image. The details may then be transferred to the visible lightimages using a dual bilateral filter as described, for example, in EricP. Bennett et al., “Multispectral Video Fusion,” Computer Graphics (ACMSIGGRAPH Proceedings) (Jul. 25, 2006), which is incorporated byreference herein in its entirety.

Dynamic Range Determination by Differing Exposures at Imagers

An auto-exposure (AE) algorithm is important to obtaining an appropriateexposure for the scene to be captured. The design of the AE algorithmaffects the dynamic range of captured images. The AE algorithmdetermines an exposure value that allows the acquired image to fall inthe linear region of the camera array's sensitivity range. The linearregion is preferred because a good signal-to-noise ratio is obtained inthis region. If the exposure is too low, the picture becomesunder-saturated while if the exposure is too high the picture becomesover-saturated. In conventional cameras, an iterative process is takento reduce the difference between measured picture brightness andpreviously defined brightness below a threshold. This iterative processrequires a large amount of time for convergence, and sometimes resultsin an unacceptable shutter delay.

In one embodiment, the picture brightness of images captured by aplurality of imagers is independently measured. Specifically, aplurality of imagers are set to capturing images with differentexposures to reduce the time for computing the adequate exposure. Forexample, in a camera array with 5×5 imagers where 8 luma imagers and 9near-IR imagers are provided, each of the imagers may be set withdifferent exposures. The near-IR imagers are used to capture low-lightaspects of the scene and the luma imagers are used to capture the highillumination aspects of the scene. This results in a total of 17possible exposures. If exposure for each imager is offset from anadjacent imager by a factor of 2, for example, a maximum dynamic rangeof 2¹⁷ or 102 dB can be captured. This maximum dynamic range isconsiderably higher than the typical 48 dB attainable in a conventionalcamera with 8 bit image outputs.

At each time instant, the responses (under-exposed, over-exposed oroptimal) from each of the multiple imagers are analyzed based on howmany exposures are needed at the subsequent time instant. The ability toquery multiple exposures simultaneously in the range of possibleexposures accelerates the search compared to the case where only oneexposure is tested at once. By reducing the processing time fordetermining the adequate exposure, shutter delays and shot-to-shot lagsmay be reduced.

In one embodiment, the HDR image is synthesized from multiple exposuresby combining the images after linearizing the imager response for eachexposure. The images from the imagers may be registered before combiningto account for the difference in the viewpoints of the imagers.

In one embodiment, at least one imager includes HDR pixels to generateHDR images. HDR pixels are specialized pixels that capture high dynamicrange scenes. Although HDR pixels show superior performances compared toother pixels, HDR pixels show poor performance at low lightingconditions in comparison with near-IR imagers. To improve performance atlow lighting conditions, signals from the near-IR imagers may be used inconjunction with the signal from the HDR imager to attain better qualityimages across different lighting conditions.

In one embodiment, an HDR image is obtained by processing imagescaptured by multiple imagers by processing, as disclosed, for example,in Paul Debevec et al., “Recovering High Dynamic Range Radiance Mapsfrom Photographs,” Computer Graphics (ACM SIGGRAPH Proceedings), (Aug.16, 1997), which is incorporated by reference herein in its entirety.The ability to capture multiple exposures simultaneously using theimager is advantageous because artifacts caused by motion of objects inthe scene can be mitigated or eliminated.

Hyperspectral Imaging by Multiple Imagers

In one embodiment, a multi-spectral image is rendered by multipleimagers to facilitate the segmentation or recognition of objects in ascene. Because the spectral reflectance coefficients vary smoothly inmost real world objects, the spectral reflectance coefficients may beestimated by capturing the scene in multiple spectral dimensions usingimagers with different color filters and analyzing the captured imagesusing Principal Components Analysis (PCA).

In one embodiment, half of the imagers in the camera array are devotedto sampling in the basic spectral dimensions (R, G, and B) and the otherhalf of the imagers are devoted to sampling in a shifted basic spectraldimensions (R′, G′, and B′). The shifted basic spectral dimensions areshifted from the basic spectral dimensions by a certain wavelength(e.g., 10 nm).

In one embodiment, pixel correspondence and non-linear interpolation isperformed to account for the sub-pixel shifted views of the scene. Thenthe spectral reflectance coefficients of the scene are synthesized usinga set of orthogonal spectral basis functions as disclosed, for example,in J. P. S. Parkkinen, J. Hallikainen and T. Jaaskelainen,“Characteristic Spectra of Munsell Colors,” J. Opt. Soc. Am., A 6:318(August 1989), which is incorporated by reference herein in itsentirety. The basis functions are eigenvectors derived by PCA of acorrelation matrix and the correlation matrix is derived from a databasestoring spectral reflectance coefficients measured by, for example,Munsell color chips (a total of 1257) representing the spectraldistribution of a wide range of real world materials to reconstruct thespectrum at each point in the scene.

At first glance, capturing different spectral images of the scenethrough different imagers in the camera array appears to traderesolution for higher dimensional spectral sampling. However, some ofthe lost resolution may be recovered. The multiple imagers sample thescene over different spectral dimensions where each sampling grid ofeach imager is offset by a sub-pixel shift from the others. In oneembodiment, no two sampling grid of the imager overlap. That is, thesuperposition of all the sampling grids from all the imagers forms adense, possibly non-uniform, montage of points. Scattered datainterpolation methods may be used to determine the spectral density ateach sample point in this non-uniform montage for each spectral image,as described, for example, in Shiaofen Fang et al., “Volume MorphingMethods for Landmark Based 3D Image Deformation” by SPIE vol. 2710,proc. 1996 SPIE Intl Symposium on Medical Imaging, page 404-415, NewportBeach, Calif. (February 1996), which is incorporated by reference hereinin its entirety. In this way, a certain amount of resolution lost in theprocess of sampling the scene using different spectral filters may berecovered.

As described above, image segmentation and object recognition arefacilitated by determining the spectral reflectance coefficients of theobject. The situation often arises in security applications wherein anetwork of cameras is used to track an object as it moves from theoperational zone of one camera to another. Each zone may have its ownunique lighting conditions (fluorescent, incandescent, D65, etc.) thatmay cause the object to have a different appearance in each imagecaptured by different cameras. If these cameras capture the images in ahyper-spectral mode, all images may be converted to the same illuminantto enhance object recognition performance.

In one embodiment, camera arrays with multiple imagers are used forproviding medical diagnostic images. Full spectral digitized images ofdiagnostic samples contribute to accurate diagnosis because doctors andmedical personnel can place higher confidence in the resultingdiagnosis. The imagers in the camera arrays may be provided with colorfilters to provide full spectral data. Such camera array may beinstalled on cell phones to capture and transmit diagnostic informationto remote locations as described, for example, in Andres W. Martinez etal., “Simple Telemedicine for Developing Regions: Camera Phones andPaper-Based Microfluidic Devices for Real-Time, Off-Site Diagnosis,”Analytical Chemistry (American Chemical Society) (Apr. 11, 2008), whichis incorporated by reference herein in its entirety. Further, the cameraarrays including multiple imagers may provide images with a large depthof field to enhance the reliability of image capture of wounds, rashes,and other symptoms.

In one embodiment, a small imager (including, for example, 20-500pixels) with a narrow spectral bandpass filters is used to produce asignature of the ambient and local light sources in a scene. By usingthe small imager, the exposure and white balance characteristics may bedetermined more accurately at a faster speed. The spectral bandpassfilters may be ordinary color filters or diffractive elements of abandpass width adequate to allow the number of camera arrays to coverthe visible spectrum of about 400 nm. These imagers may run at a muchhigher frame rate and obtain data (which may or may not be used for itspictorial content) for processing into information to control theexposure and white balance of other larger imagers in the same cameraarray. The small imagers may also be interspersed within the cameraarray.

Optical Zoom Implemented Using Multiple Imagers

In one embodiment, a subset of imagers in the camera array includestelephoto lenses. The subset of imagers may have other imagingcharacteristics that are the same as imagers with non-telephoto lenses.Images from this subset of imagers are combined and super-resolutionprocessed to form a super-resolution telephoto image. In anotherembodiment, the camera array includes two or more subsets of imagersequipped with lenses of more than two magnifications to providediffering zoom magnifications.

Embodiments of the camera arrays may achieve its final resolution byaggregating images through super-resolution. Taking an example ofproviding 5×5 imagers with a 3× optical zoom feature, if 17 imagers areused to sample the luma (G) and 8 imagers are used to sample the chroma(R and B), 17 luma imagers allow a resolution that is four times higherthan what is achieved by any single imager in the set of 17 imagers. Ifthe number of the imagers is increased from 5×5 to 6×6, an addition of11 extra imagers becomes available. In comparison with the 8 Megapixelconventional image sensor fitted with a 3× zoom lens, a resolution thatis 60% of the conventional image sensor is achieved when 8 of theadditional 11 imagers are dedicated to sampling luma (G) and theremaining 3 imagers are dedicated to chroma (R and B) and near-IRsampling at 3× zoom. This considerably reduces the chroma sampling (ornear-IR sampling) to luma sampling ratio. The reduced chroma to lumasampling ratio is somewhat offset by using the super-resolved luma imageat 3× zoom as a recognition prior on the chroma (and near-IR) image toresample the chroma image at a higher resolution.

With 6×6 imagers, a resolution equivalent to the resolution ofconventional image sensor is achieved at 1× zoom. At 3× zoom, aresolution equivalent to about 60% of conventional image sensoroutfitted with a 3× zoom lens is obtained by the same imagers. Also,there is a decrease in luma resolution at 3× zoom compared withconventional image sensors with resolution at 3× zoom. The decreasedluma resolution, however, is offset by the fact that the optics ofconventional image sensor has reduced efficiency at 3× zoom due tocrosstalk and optical aberrations.

The zoom operation achieved by multiple imagers has the followingadvantages. First, the quality of the achieved zoom is considerablyhigher than what is achieved in the conventional image sensor due to thefact that the lens elements may be tailored for each change in focallength. In conventional image sensors, optical aberrations and fieldcurvature must be corrected across the whole operating range of thelens, which is considerably harder in a zoom lens with moving elementsthan in a fixed lens element where only aberrations for a fixed focallength need to be corrected. Additionally, the fixed lens in the imagershas a fixed chief ray angle for a given height, which is not the casewith conventional image sensor with a moving zoom lens. Second, theimagers allow simulation of zoom lenses without significantly increasingthe optical track height. The reduced height allows implementation ofthin modules even for camera arrays with zooming capability.

The overhead required to support a certain level of optical zoom incamera arrays according to some embodiments is tabulated in Table 2.

TABLE 2 No. of Luma No. of Imagers at No. of Chroma Imagers in differentZoom Imagers at different Camera levels Zoom Levels array 1X 2X 3X 1X 2X3X 25 17 0 0 8 0 0 36 16 0 8 8 0 4

In one embodiment, the pixels in the images are mapped onto an outputimage with a size and resolution corresponding to the amount of zoomdesired in order to provide a smooth zoom capability from thewidest-angle view to the greatest-magnification view. Assuming that thehigher magnification lenses have the same center of view as the lowermagnification lenses, the image information available is such that acenter area of the image has a higher resolution available than theouter area. In the case of three or more distinct magnifications, nestedregions of different resolution may be provided with resolutionincreasing toward the center.

An image with the most telephoto effect has a resolution determined bythe super-resolution ability of the imagers equipped with the telephotolenses. An image with the widest field of view can be formatted in atleast one of two following ways. First, the wide field image may beformatted as an image with a uniform resolution where the resolution isdetermined by the super-resolution capability of the set of imagershaving the wider-angle lenses. Second, the wide field image is formattedas a higher resolution image where the resolution of the central part ofthe image is determined by the super-resolution capability of the set ofimagers equipped with telephoto lenses. In the lower resolution regions,information from the reduced number of pixels per image area isinterpolated smoothly across the larger number of “digital” pixels. Insuch an image, the pixel information may be processed and interpolatedso that the transition from higher to lower resolution regions occurssmoothly.

In one embodiment, zooming is achieved by inducing a barrel-likedistortion into some, or all, of the array lens so that adisproportionate number of the pixels are dedicated to the central partof each image. In this embodiment, every image has to be processed toremove the barrel distortion. To generate a wide-angle image, pixelscloser to the center are sub-sampled relative to outer pixels aresuper-sampled. As zooming is performed, the pixels at the periphery ofthe imagers are progressively discarded and the sampling of the pixelsnearer the center of the imager is increased.

In one embodiment, mipmap filters are built to allow images to berendered at a zoom scale that is between the specific zoom range of theoptical elements (e.g., 1× and 3× zoom scales of the camera array).Mipmaps are a precalculated optimized set of images that accompany abaseline image. A set of images associated with the 3× zoom luma imagecan be created from a baseline scale at 3× down to 1×. Each image inthis set is a version of the baseline 3× zoom image but at a reducedlevel of detail. Rendering an image at a desired zoom level is achievedusing the mipmap by (i) taking the image at 1× zoom, and computing thecoverage of the scene for the desired zoom level (i.e., what pixels inthe baseline image needs to be rendered at the requested scale toproduce the output image), (ii) for each pixel in the coverage set,determine if the pixel is in the image covered by the 3× zoom lumaimage, (iii) if the pixel is available in the 3× zoom luma image, thenchoose the two closest mipmap images and interpolate (using smoothingfilter) the corresponding pixels from the two mipmap images to producethe output image, and (iv) if the pixel is unavailable in the 3× zoomluma image, then choose the pixel from the baseline 1× luma image andscale up to the desired scale to produce the output pixel. By usingmipmaps, smooth optical zoom may be simulated at any point between twogiven discrete levels (i.e., 1× zoom and 3× zoom).

In one embodiment, zooming is achieved by realizing different Fields OfView (FOV)s by electronically switching between different opticalchannels having different sensor sizes, but fixed Effective FocalLengths (EFL)s. In one such embodiment, shown schematically in FIG. 8A,variable FOVs are achieved by creating optical channels on the samesubstrate that have different imager sizes 800 and 802 at the same fixedEFL 804. Using this structure, it would be possible to create anarbitrary number of zoom magnifications by including image sensors withlarger or smaller numbers of pixels. This technique is particularlysimple to incorporate into WLO array cameras as these variable zoomsensor arrays 800 and 802 can be fabricated directly onto the basecamera array substrate without any further modification to the design ofthe array camera assembly itself.

In another embodiment, as shown in FIG. 8B, different FOVs are achievedby engineering different EFLs 805 into specific optical channels of thecamera array 806 while maintaining a fixed imager size 808. Implementingdifferent EFLs on the same substrate stacks, i.e., substrate stacks withconstant thicknesses and spacings, is more complicated, because thedistance of the principal plane and with it the entrance pupil andconsequently the aperture stop 810 with respect to the image sensor 814needs to be changed in order to change the focal length of the opticalchannel. In the current embodiment, this is accomplished by theintroduction of “dummy substrates” 816, 818 and 820 into the stack 806,such that each zoom channel 822, 824 and 826 has an associated aperturestop 828, 830 and 832 disposed on a different substrate or a differentface of a substrate, such that different EFLs can be achieved. As shown,while the distribution and positioning of the lenses (834, 836 and 838)and the aperture stops (828, 830 and 832) on the particular substrate orsubstrate face is entirely dependent on the desired EFL, in all casesthe substrate thicknesses and distances remain fixed. Alternatively, insuch an embodiment, each of the substrates could be provided withlenses, but in different distributions in order to allow for differentEFLs. Such a structure would allow for higher image quality, but athigher cost.

In another further embodiment, as shown in FIG. 8C, different FOVs canalso be achieved by engineering different EFLs 805 using “dummy”substrates in a manner similar to that illustrated in FIG. 8B, with theexception that all of the substrates have lenses elements on them ineach optical channel. The lens elements, however, have differentprescriptions in order to allow different EFLs. Accordingly, any of avariety of configurations of optics and sensor size, and/or lightsensing element size can be utilized with an array camera in accordancewith embodiments of the invention to achieve different FOVs.

Capturing Video Images

In one embodiment, the camera array generates high frame imagesequences. The imagers in the camera array can operate independently tocapture images. Compared to conventional image sensors, the camera arraymay capture images at the frame rate up to N time (where N is the numberof imagers). Further, the frame period for each imager may overlap toimprove operations under low-light conditions. To increase theresolution, a subset of imagers may operate in a synchronized manner toproduce images of higher resolution. In this case, the maximum framerate is reduced by the number of imagers operating in a synchronizedmanner. The high-speed video frame rates can enables slow-motion videoplayback at a normal video rate.

In one example, two luma imagers (green imagers or near-IR imagers), twoblue imagers and two green imagers are used to obtain high-definition1080p images. Using permutations of four luma imagers (two green imagersand two near-IR imagers or three green imagers and one near-IR imager)together with one blue imager and one red imager, the chroma imagers canbe upsampled to achieve 120 frames/sec for 1080p video. For higher framerate imaging devices, the number of frame rates can be scaled uplinearly. For Standard-Definition (480p) operation, a frame rate of 240frames/sec may be achieved using the same camera array.

Conventional imaging devices with a high-resolution image sensor (e.g.,8 Megapixels) use binning or skipping to capture lower resolution images(e.g., 1080p30, 720p30 and 480p30). In binning, rows and columns in thecaptured images are interpolated in the charge, voltage or pixel domainsin order to achieve the target video resolutions while reducing thenoise. In skipping, rows and columns are skipped in order to reduce thepower consumption of the sensor. Both of these techniques result inreduced image quality.

In one embodiment, the imagers in the camera arrays are selectivelyactivated to capture a video image. For example, 9 imagers (includingone near-IR imager) may be used to obtain 1080p (1920×1080 pixels)images while 6 imagers (including one near-IR imager) may be used toobtain 720p (1280×720 pixels) images or 4 imagers (including one near-IRimager) may be used to obtain 480p (720×480 pixels) images. Becausethere is an accurate one-to-one pixel correspondence between the imagerand the target video images, the resolution achieved is higher thantraditional approaches. Further, since only a subset of the imagers isactivated to capture the images, significant power savings can also beachieved. For example, 60% reduction in power consumption is achieved in1080p and 80% of power consumption is achieved in 480p.

Using the near-IR imager to capture video images is advantageous becausethe information from the near-IR imager may be used to denoise eachvideo image. In this way, the camera arrays of embodiments exhibitexcellent low-light sensitivity and can operate in extremely low-lightconditions. In one embodiment, super-resolution processing is performedon images from multiple imagers to obtain higher resolution videoimagers. The noise-reduction characteristics of the super-resolutionprocess along with fusion of images from the near-IR imager results in avery low-noise images.

In one embodiment, high-dynamic-range (HDR) video capture is enabled byactivating more imagers. For example, in a 5×5 camera array operating in1080p video capture mode, there are only 9 cameras active. A subset ofthe 16 cameras may be overexposed and underexposed by a stop in sets oftwo or four to achieve a video output with a very high dynamic range.

Other Applications for Multiple Imagers

In one embodiment, the multiple imagers are used for estimating distanceto an object in a scene. Since information regarding the distance toeach point in an image is available in the camera array along with theextent in x and y coordinates of an image element, the size of an imageelement may be determined. Further, the absolute size and shape ofphysical items may be measured without other reference information. Forexample, a picture of a foot can be taken and the resulting informationmay be used to accurately estimate the size of an appropriate shoe.

In one embodiment, reduction in depth of field is simulated in imagescaptured by the camera array using distance information. The cameraarrays according to the present invention produce images with greatlyincreased depth of field. The long depth of field, however, may not bedesirable in some applications. In such case, a particular distance orseveral distances may be selected as the “in best focus” distance(s) forthe image and based on the distance (z) information from parallaxinformation, the image can be blurred pixel-by-pixel using, for example,a simple Gaussian blur. In one embodiment, the depth map obtained fromthe camera array is utilized to enable a tone mapping algorithm toperform the mapping using the depth information to guide the level,thereby emphasizing or exaggerating the 3D effect.

In one embodiment, apertures of different sizes are provided to obtainaperture diversity. The aperture size has a direct relationship with thedepth of field. In miniature cameras, however, the aperture is generallymade as large as possible to allow as much light to reach the cameraarray. Different imagers may receive light through apertures ofdifferent sizes. For imagers to produce a large depth of field, theaperture may be reduced whereas other imagers may have large aperturesto maximize the light received. By fusing the images from sensor imagesof different aperture sizes, images with large depth of field may beobtained without sacrificing the quality of the image.

In one embodiment, the camera array according to the present inventionrefocuses based on images captured from offsets in viewpoints. Unlike aconventional plenoptic camera, the images obtained from the camera arrayof the present invention do not suffer from the extreme loss ofresolution. The camera array according to the present invention,however, produces sparse data points for refocusing compared to theplenoptic camera. In order to overcome the sparse data points,interpolation may be performed to refocus data from the spare datapoints.

In one embodiment, each imager in the camera array has a differentcentroid. That is, the optics of each imager are designed and arrangedso that the fields of view for each imager slightly overlap but for themost part constitute distinct tiles of a larger field of view. Theimages from each of the tiles are panoramically stitched together torender a single high-resolution image.

In one embodiment, camera arrays may be formed on separate substratesand mounted on the same motherboard with spatial separation. The lenselements on each imager may be arranged so that the corner of the fieldof view slightly encompasses a line perpendicular to the substrate.Thus, if four imagers are mounted on the motherboard with each imagerrotated 90 degrees with respect to another imager, the fields of viewwill be four slightly overlapping tiles. This allows a single design ofWLO lens array and imager chip to be used to capture different tiles ofa panoramic image.

In one embodiment, one or more sets of imagers are arranged to captureimages that are stitched to produce panoramic images with overlappingfields of view while another imager or sets of imagers have a field ofview that encompasses the tiled image generated. This embodimentprovides different effective resolution for imagers with differentcharacteristics. For example, it may be desirable to have more luminanceresolution than chrominance resolution. Hence, several sets of imagersmay detect luminance with their fields of view panoramically stitched.Fewer imagers may be used to detect chrominance with the field of viewencompassing the stitched field of view of the luminance imagers.

In one embodiment, the camera array with multiple imagers is mounted ona flexible motherboard such that the motherboard can be manually bent tochange the aspect ratio of the image. For example, a set of imagers canbe mounted in a horizontal line on a flexible motherboard so that in thequiescent state of the motherboard, the fields of view of all of theimagers are approximately the same. If there are four imagers, an imagewith double the resolution of each individual imager is obtained so thatdetails in the subject image that are half the dimension of details thatcan be resolved by an individual imager. If the motherboard is bent sothat it forms part of a vertical cylinder, the imagers point outward.With a partial bend, the width of the subject image is doubled while thedetail that can be resolved is reduced because each point in the subjectimage is in the field of view of two rather than four imagers. At themaximum bend, the subject image is four times wider while the detailthat can be resolved in the subject is further reduced.

Offline Reconstruction and Processing

The images processed by the imaging system 400 may be previewed beforeor concurrently with saving of the image data on a storage medium suchas a flash device or a hard disk. In one embodiment, the images or videodata includes rich light field data sets and other useful imageinformation that were originally captured by the camera array. Othertraditional file formats could also be used. The stored images or videomay be played back or transmitted to other devices over various wired orwireless communication methods.

In one embodiment, tools are provided for users by a remote server. Theremote server may function both as a repository and an offlineprocessing engine for the images or video. Additionally, applets mashedas part of popular photo-sharing communities such as Flikr, Picasaweb,Facebook etc. may allow images to be manipulated interactively, eitherindividually or collaboratively. Further, software plug-ins into imageediting programs may be provided to process images generated by theimaging device 400 on computing devices such as desktops and laptops.

Various modules described herein may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers referred to in the specification may include a singleprocessor or may be architectures employing multiple processor designsfor increased computing capability.

While particular embodiments and applications of the present inventionhave been illustrated and described herein, it is to be understood thatthe invention is not limited to the precise construction and componentsdisclosed herein and that various modifications, changes, and variationsmay be made in the arrangement, operation, and details of the methodsand apparatuses of the present invention without departing from thespirit and scope of the invention as it is defined in the appendedclaims.

What is claimed is:
 1. A camera array, comprising: a plurality ofcameras configured to capture images of a scene, where each cameracomprises: optics comprising at least one lens element and at least oneaperture; and a sensor comprising a two-dimensional array of pixels andcontrol circuitry for controlling imaging parameters; a processorconfigured by software to: capture a plurality of images from differentviewpoints using the plurality of cameras, where each image captured bythe plurality of cameras includes pixels that are occluded in at leastone other image captured by the plurality of cameras; and normalize theplurality of images based upon calibration data to enable scan-linebased parallax searches; measure parallax between the normalized imagesby adaptively comparing the similarity of neighborhoods of pixels fordifferent parallax-induced shifts along scan-lines; identify occludedpixels based upon the measured parallax information; generate a depthmap using the measured parallax information; select at least onedistance as an “in best focus” distance; and blur an image produced bythe camera array based upon the “in best focus” distance and distanceinformation from the depth map.
 2. The camera array of claim 1, whereinthe plurality of cameras comprises an array of camera arrays.
 3. Thecamera array of claim 1, wherein the plurality of cameras are arrangedas a first set of cameras forming a first camera array, and a second setof cameras forming a second camera array.
 4. The camera array of claim1, wherein at least two of the plurality of cameras are separated by adistance approximating the separation of human eyes.
 5. The camera arrayof claim 1, wherein each camera includes a spectral filter configured topass a specific spectral band of light selected from the groupconsisting of a Bayer filter, one or more Blue filters, one or moreGreen filters, one or more Red filters, one or more shifted spectralfilters, one or more near-IR filters, and one or more hyper-spectralfilters.
 6. The camera array of claim 1, wherein the control circuitryin each of plurality of cameras is capable of configuring the pluralityof cameras to operate with at least one difference in operatingparameters.
 7. The camera array of claim 6, wherein the at least onedifference in operating parameters includes at least one imagingparameter selected from the group consisting of exposure time, gain, andblack level offset.
 8. The camera array of claim 6, wherein the cameraarray is a monolithic camera array assembly comprising: a lens elementarray forming the optics of each camera; and a single semiconductorsubstrate on which all of the pixels and control circuitry for eachcamera are formed.
 9. The camera array of claim 1, wherein the pluralityof cameras are formed on separate semiconductor substrates.
 10. Thecamera array of claim 9, wherein the plurality of cameras are mounted toa printed circuit board.
 11. The camera array of claim 1, wherein atleast two cameras have different focal lengths.
 12. The camera array ofclaim 1, wherein the sensor in each of at least two cameras havedifferent resolutions.
 13. The camera array of claim 1, wherein thesensor in each of at least two cameras have different sizes.
 14. Thecamera array of claim 1, wherein the processor is further configured bysoftware to measure parallax by determining the parallax that yields thehighest correlation between pixels from images captured by the pluralityof cameras accounting for the positions of the cameras that captured theimages.
 15. The camera array of claim 1, wherein the processor isfurther configured by software to perform pair-wise measurements ofneighborhoods of pixels to determine pixel similarity for differentparallax-induced shifts.
 16. The camera array of claim 15, wherein theprocessor is further configured by software to determine a parallax thatyields a highest similarity between pixels from images captured bykeeping track of various pair-wise measurements of neighborhoods ofpixels and calculating a parallax that yields the highest similarity asthe best least squares fit of the pair-wise measurements ofneighborhoods of pixels.
 17. The camera array of claim 1, wherein theprocessor is further configured by software to generate at least oneimage by fusing aligned portions of the plurality of captured imagesusing the depth map.
 18. The camera array of claim 17, wherein theprocessor is further configured by software to perform super-resolutionprocessing on the fused image portions to synthesize a super-resolutionimage.
 19. The camera array of claim 1, wherein the processor is furtherconfigured by software to generate a higher resolution super-resolvedimage synthesized using the plurality of images and the parallaxmeasurements to compensate for parallax in the plurality of images. 20.A camera array, comprising: a plurality of cameras configured to captureimages of a scene, where each camera comprises: optics comprising atleast one lens element and at least one aperture; and a sensorcomprising a two-dimensional array of pixels and control circuitry forcontrolling imaging parameters; wherein at least one of the plurality ofcameras comprises a Bayer filter and the control circuitry in each ofplurality of cameras is capable of configuring the plurality of camerasto operate with at least one difference in operating parameters; aprocessor configured by software to: capture a plurality of images fromdifferent viewpoints using the plurality of cameras, where each imagecaptured by the plurality of cameras includes pixels that are occludedin at least one other image captured by the plurality of cameras; andnormalize the plurality of images based upon calibration data to enablescan-line based parallax searches; measure parallax between thenormalized images by adaptively comparing the similarity ofneighborhoods of pixels for different parallax-induced shifts alongscan-lines by performing pair-wise measurements of neighborhoods ofpixels to determine pixel similarity for different parallax-inducedshifts; identify occluded pixels based upon the measured parallaxinformation; generate a depth map using the measured parallaxinformation by discarding measured parallax information determined usingoccluded pixels; select at least one distance as an “in best focus”distance; and blur an image produced by the camera array based upon the“in best focus” distance and distance information from the depth map.